Orphaned reply pidboxes in Redis

[rkul]

Hello all
We use Celery 3.0.21 with Redis backend (AWS ElastiCache) and recently I discovered huge memory usage by Redis node. After dumping Redis database and inspecting it I found that there are many reply pidboxes (~40) with 1 big element in array (about 16MB) in which there are many duplicated scheduled tasks (more than thousand).

I think duplicated tasks is a different issue. So could anybody clarifing approach to delete unused reply pidboxes? I've checked number of reply pidboxes with members of _kombu.binding.reply.celery.pidbox hash and found inconsistency (41 in bindings versus 77 at all).

And I would much appreciate it if anybody explaine case when these orphaned pidboxes could appear.

[ask]

These replies are sent for remote control commands. I know flower is constantly sending out control commands, but I'm not sure there is an easy fix.

The best option would be if the replies were using PUBSUB instead of being stored in a list.

cc @mher

[31z4]

Are these replies safe to delete @ask?

[tilgovi]

I had the same problem. I disabled gossip and mingle to prevent these from getting created. I found that these were growing larger and larger due to an ever-growing list of revoked tasks in the mingle replies.

[michi88]

For my future self and other googlers. This can be disabled in django with:

CELERY_WORKER_ENABLE_REMOTE_CONTROL = False # celery 4
CELERY_ENABLE_REMOTE_CONTROL = False # celery 3

see: https://docs.celeryproject.org/en/stable/userguide/configuration.html#worker-enable-remote-control

And yes, had this problem twice because of the name change from v3 to v4 :(