Rate limit tasks by user

[snarfed]

We've started seeing sustained spikes of incoming deletes from ATProto recently, all roughly 2/s. This backs up send a lot, often as much as 15-30m. We saw similar unfollow floods a while ago from follow spammers, #1669, which we blocked, but I think these are deletes, not unfollows. Need to investigate more.

[snarfed]

Current instance is almost all from one user, did:plc:v35n6kafnti65tv44uuwtbmn. Example: at://did:plc:v35n6kafnti65tv44uuwtbmn/app.bsky.feed.post/3lcvbt6idwm2w#delete

{
  "objectType": "activity",
  "verb": "delete"
  "id": "at://did:plc:v35n6kafnti65tv44uuwtbmn/app.bsky.feed.post/3lcvbt6idwm2w#delete",
  "actor": "did:plc:v35n6kafnti65tv44uuwtbmn",
  "object": "at://did:plc:v35n6kafnti65tv44uuwtbmn/app.bsky.feed.post/3lcvbt6idwm2w",
}

Looks like a valid delete of a valid post that we bridged back in Dec.

[snarfed]

This one at least seems like someone mass deleting their old posts, which we probably want to allow, and bridge. So, maybe we rate limit per user and gradually spread out their receive and send tasks into the future when they go over the rate limit?

[snarfed]

A fun bit here will be figuring out how to implement eg leaky bucket on memcache, where we have a limited set of operations. And also where the result isn't blocking, but pushing task ETAs out farther into the future based on a user's task volume.

[snarfed]

...or we do it probabilistically! https://blog.cloudflare.com/sometimes-i-cache was great recent read on this.

[snarfed]

I'm going to do this as part of the fix for #2089. That means I need to spread out receive tasks deterministically, always, so I'm not going to do leaky bucket or probabilistic. Instead, I'm thinking:

• When we enqueue a new receive task, get our current delay for its user from memcache, add 5s, and use that as the task's ETA.
• If its user has no current delay, set it to 0s.
• Set the memcache row to expire at its delay timestamp.

This will run each user's receive tasks at a constant rate of one every 5s.

cc @anujahooja for the product/UX implications of this. (We run a receive task to handle each unique incoming activity, eg new post, like, repost, delete, follow, unfollow, etc.)

[snarfed]

Shipped this ^, it's working! Probably need to tweak delay monitoring calculation a bit, but we've gone from steady state of receive task queue at roughly zero to many thousand. Spot checked, they all seem legitimately rate limited. 🤞🤞🤞

[snarfed]

Love seeing these shapes. Spikes of enqueued receive tasks, processed gradually over time, no visible corresponding spikes in task run rate.