Cleanup of postgres based messaging blocks workers from actually executing other tasks

[ruifung]

Describe the bug

It seems like across 3 worker instances, they seem to not be efficiently executing tasks? I've observed from my postgres database a bunch of connections executing what seems to be the same delete statement.

This seems like really inefficient behavior? It seems like multiple "remove expired objects" tasks are in the task queue and you can apparently get 3 workers picking up the same job, which is really inefficient.

And I'm pretty sure this is a complete waste of my self hosted compute resources since the task queue is slowly increasing while it looks like 3 workers are contenting with each other to delete the same things.

How to reproduce

As far as I'm aware, it's just my installation with multiple workers, and admittedly, not a very fast database.

I've just noticed when checking the active connections from authentik in my postgres installation.

Expected behavior

I expect the workers to not assign ALL the workers simultaneously to trying to clean up expired objects. What's the point of running multiple workers when they can get stuck doing the same thing together.

Also, deleting each message row individually is really bad when apparently somehow on my installation it's generating, on average, a thousand messages per minute somehow. (Although a brief look through that table makes it look like it's all identical messages?)

Screenshots

Also look at all of them blocking on each other when I insert my own query to mass-delete expired messages.

Additional context

Also, I've observed authentik (collectively) holding a significant number of idle connections over time in my postgres DB, as such, I actually enabled idle session timeout and idle in transaction session timeout on my database (It was literally kneeling over from authentik holding 140+ connections when I had a 200 connection limit).

Also there are 7,860,119 mesasges in that django_channels_postgres_message table (I just truncated it a few hours ago!), that the 3 workers seem to be clearing very inefficiently. Since all of them seem to be attempting to delete them one at a time.. on the same message across 3 workers.

These instances are all connecting directly to the primary postgres instance, with no connection pooler in between.

For now I've somewhat mitigated it by using pg_cron to run a pair of cleanup queries specifically targeting the django_channels_postgres_message and django_channels_postgres_groupchannel tables to remove expired entries.

Deployment Method

Kubernetes

Version

2026.2.0

Relevant log output

[ruifung]

Looking at the code in

authentik/authentik/core/tasks.py

Line 42 in 3d28439

obj.delete()

It seems like it is doing deletion of expired messages and groups, individually, issuing deletes one at a time. This is supremely inefficient as it appears to be (on postgres at least) issuing individual Delete queries for every single expired message. (As opposed to just doing a bulk delete with condition).

Is there any reason why this is done this way? Because ever since I added my own bulk cleanup queries using pg_cron, I've noticed that my workers are actually able to process the other tasks now, instead of having all of the processing separate cleanup tasks, where they all keep looping over the same data (because all of them queried in separate transactions, and is deleting the same messages in separate workers)

Looking at the django docs, it appears they support bulk deleting of an entire queryset?

    for cls in [Message, GroupChannel]:
        objects = cls.objects.all().filter(expires__lt=now())
        amount = objects.count()
        objects.delete()

[ruifung]

I would say that there are probably 2 issues here:

1. If message creation exceeds the ability of the workers to delete individually, it can never clean the tables fast enough.
2. Multiple tasks for the cleanup can be issued over time, and if a worker is stuck handling (1), multiple workers can all end up stuck handling (1) because they run in -separate transactions- on postgresql and have no visibility into each other deleting, and possibly, start looping over the same data, and basically slowing down the process even more because of postgres side locks.
3. As a result of (2), the system queued tasks can continue to pile up, with potentially even more cleanup tasks in the log, resulting in system task processing basically halting, and constantly compounding into a state where manual intervention (truncating the tasks table) and manual cleanup of the messages table is needed.

Additional context:
It seems message activity occurs in spikes for some reason, even on my small installation, and coupled with the above issues, can, and has snowballed hard.

[rissson]

Are you using any RAC provider?

[ruifung]

No, the only providers I have configured are LDAP, SAML, OIDC, and Proxy (Forward Auth)

[rissson]

Interesting, usually it's the RAC provider that is most chatty in websocket messages. I'll try and whip up something to delete messages earlier.