this seems a bit unbelievable for some reason?

Am wondering if it was actually the AP stuff that was the burden, or whether it was the thundering herd requesting the link previews?

I just built a whole thing in Ghost specifically because it was moving toward AP integration and this is…alarming, especially since Ghost plans to charge for fedi subscribers. (Which, no wonder, given that situation.)

But like: WHAT?

@thisismissem @bengo Yup! We've got to figure some things out here for sure.

You're absolutely right in your assessment that some of the work is on the side of our (fledgling) implementation (eg. queues) and some may also be needed at the protocol level.

cc @evanprodromou

@Elena Rossini ⁂

Elena,
I think you could be making a mistake here. If I was a newsletter recipient, I'd be doing everything in my power to prevent any indication of opening or interaction.

@thisismissem also worth noting ghost is built in nodejs, so it's more or less single threaded. 10 servers might very well have been 10 cpu cores in a different stack.

Anyway, this is to say that activitypub is very resource intensive, but this seems like there are complicating factors that can be worked through over time.

@jenniferplusplus
True, node.js is single threaded, however due to the async i/o you can usually process a fair number of requests simultaneously, because a single request doesn't have to finish before another is processed — you only get into trouble with synchronous APIs and like long-running processing (e.g., iterating over a lot of data)

I suspect besides queuing, there's something non-obvious here.

@thisismissem

Not using a queue is… certainly a choice one can make.

@jenniferplusplus @kissane @fediversereport

@jenniferplusplus

Absolutely. It also isn't particularly complicated to add a queue, albeit significantly easier earlier in the dev process, but the logistics of distributing a queue can be much more nuanced.

This does, however, raise questions for me about what their goals are here.

@thisismissem @kissane @fediversereport

@thisismissem

I'm not sure that's actually a useful thing to analyze for these sorts of questions? Like I'd be interested in it regardless, but the reason you generally have a queue is not performance.

(Unless by "benchmark" it means "throughput benchmarking" in which case it is useful but it is really just benchmarking the queue's performance and is highly sensitive to it)

@jenniferplusplus @kissane @fediversereport @hongminhee

@thisismissem

But even if it lowered the throughput, it's still not done generally for that reason.

You have queues to smooth out traffic spikes, to provide backpressure, and to give resiliency if a process goes down. Not really for anything you'd benchmark?

@jenniferplusplus @kissane @fediversereport @hongminhee

@thisismissem

It does, and maybe I'm misunderstanding, I'm just not convinced it matters when the reason you are doing something is not performance related in the first place.

Like NIO is classically slower than synchronous IO in java, but benefits of NIO aren't about performance but about program design and flexibility with different arch (and if the performance difference is enough to matter you probably shouldn't be using java).

@jenniferplusplus @kissane @fediversereport @hongminhee

@thisismissem

Like we'd always benchmark apps with the queue, but comparing them to the non-queue scenario seems very weird to me

Because I can't think of a reason you wouldn't already want a queue in a distributed system like this for non-performance reasons

So of course you benchmark, but only the case with a queue means anything and then you tune your queue against different scenarios, or compare types of queues. If that makes sense

@jenniferplusplus @kissane @fediversereport @hongminhee

@hrefna @jenniferplusplus @hongminhee

I think the benchmark is more to show how much faster the activitypub processing is when you defer that work by using a queue vs not, i.e., the HTTP requests duration drops significantly most likely; but autoscaling on open http requests / connections probably isn't right for node.js, instead you'd want a combination of event loop lag, mem/cpu, and open requests