Celery beat scheduler - Multiple Task Executions

[vpogrebi]

Checklist

• I have verified that the issue exists against the master branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version: 5.2.7

celery report Output:

software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.8.13
            billiard:3.6.4.0 redis:4.3.1
platform -> system:Linux arch:64bit, ELF
            kernel version:3.10.0-1160.62.1.el7.x86_64 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:redis results:disabled

deprecated_settings: None

Steps to Reproduce

1. Schedule two (or more) tasks to start at the same time
2. When one task completes, scheduler sync (tick()) is executed
3. If the other task is still running, is_due() returns True - resulting in tick() execute self.apply_entry() which essentially starts same task (the one that is already executing) to execute again

Required Dependencies

• Minimal Python Version: 3.8
• Minimal Celery Version: 5.2.7 (not necessarily minimal)
• Minimal Kombu Version: 5.2.4
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: Linux
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp @ file:///data/zkci8im/.cache/pypoetry/artifacts/22/94/d8/7787846037071cf203833b4891393d7fee750673537eccf83374f4e68a/amqp-5.1.1-py3-none-any.whl
aniso8601 @ file:///data/zkci8im/.cache/pypoetry/artifacts/cb/bb/0b/f3e74131c6390dbb7622617f9c8a590661cd4910e2e342c07ab40fe8d5/aniso8601-7.0.0-py2.py3-none-any.whl
ansible==5.1.0
ansible-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/6d/b4/59/459ce5c2e21433c1443caaecc97cbaa260085761ef58ad0794f9ef1894/ansible-core-2.12.6.tar.gz
asgiref @ file:///data/zkci8im/.cache/pypoetry/artifacts/60/d3/77/8e5b9c433ba4707af49c8a8879a4792689501bc21468d866dcba293387/asgiref-3.5.2-py3-none-any.whl
async-timeout @ file:///data/zkci8im/.cache/pypoetry/artifacts/9e/20/3e/134e90bed6cb6814ec11555be6900e30ab163641792ec5003bd6aa59d7/async_timeout-4.0.2-py3-none-any.whl
attrs @ file:///data/zkci8im/.cache/pypoetry/artifacts/4e/b8/3f/509c2f0fdcb7f29b8a253972760d0c7830bb0b0403328ae0eab3cea799/attrs-21.4.0-py2.py3-none-any.whl
billiard @ file:///data/zkci8im/.cache/pypoetry/artifacts/d7/9f/c2/e3fc46d8ea60110a6d6c6b23cabe49bb06db28d7ef3fcef9f560c37eb1/billiard-3.6.4.0-py3-none-any.whl
boto3 @ file:///data/zkci8im/.cache/pypoetry/artifacts/1d/2a/5a/b2c1f3b722c8e02330dba16637ed5b9f12d195bef6226e2d72e6112a7b/boto3-1.23.10-py3-none-any.whl
botocore @ file:///data/zkci8im/.cache/pypoetry/artifacts/10/6b/56/b501c514fffa9d9a45ecf0d334e7717c7d963cae46d6cdfd1c90d0a192/botocore-1.26.10-py3-none-any.whl
cached-property @ file:///data/zkci8im/.cache/pypoetry/artifacts/c4/6c/91/92f6a180669978ba5b5cfeea34239598e4a62f798452e667bab474eca3/cached_property-1.5.2-py2.py3-none-any.whl
celery @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/72/4d/52e0f4d27000064a2266f9d939f1a9e6e8a385310ea28ef12102c76768/celery-5.2.7-py3-none-any.whl
certifi @ file:///data/zkci8im/.cache/pypoetry/artifacts/9a/24/91/91ca3aa0f55638b8b2e2e0751f2e1cba1f342c36d4eea2028b6fc96431/certifi-2022.5.18.1-py3-none-any.whl
cffi @ file:///data/zkci8im/.cache/pypoetry/artifacts/e8/1b/a4/670c457d4f006c93740689d474309dfa24a89159e27073f4b3fc44bb98/cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
charset-normalizer @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/e6/c2/e485049fad66be34cb361166725f59e4e9ef24804c75713cbb3bd6699d/charset_normalizer-2.0.12-py3-none-any.whl
click @ file:///data/zkci8im/.cache/pypoetry/artifacts/e2/03/35/a5b93ff313b614d10ef8c1dfecb531470309e3799cbf6fbfeed0a1ce0d/click-8.1.3-py3-none-any.whl
click-didyoumean @ file:///data/zkci8im/.cache/pypoetry/artifacts/39/30/2c/da96c86e55be9b67c908b36e64e815e3c9db43db453e65b2f57ef7f52d/click_didyoumean-0.3.0-py3-none-any.whl
click-plugins @ file:///data/zkci8im/.cache/pypoetry/artifacts/d0/16/38/b7c58246dc79c9091376c792a8981f86944dd0ebe9581a58ba9a5ce0ea/click_plugins-1.1.1-py2.py3-none-any.whl
click-repl @ file:///data/zkci8im/.cache/pypoetry/artifacts/f2/c0/d5/b58b59f548e7a632f7780eae3f73f0258eb1e926d52f0170c21304ad4b/click_repl-0.2.0-py3-none-any.whl
colorama @ file:///data/zkci8im/.cache/pypoetry/artifacts/da/e1/ed/183662e38ad81210d1ff1a86d8ce34409a7d6a1f7d0ef629715dab9344/colorama-0.4.4-py2.py3-none-any.whl
coreapi @ file:///data/zkci8im/.cache/pypoetry/artifacts/e0/15/0a/3368989abeaa634a32dd92a149db85845315e1528a202d163bd8def96b/coreapi-2.3.3-py2.py3-none-any.whl
coreschema @ file:///data/zkci8im/.cache/pypoetry/artifacts/ac/20/6f/08c281ffb0c8eff16526a9b74d5af61dbcac6172ac2605f4db6c4c480a/coreschema-0.0.4.tar.gz
cryptography @ file:///data/zkci8im/.cache/pypoetry/artifacts/dd/ef/07/3dd5838cbc8f2b3a22dc4db8399410fe13a6a81f32ce2b40d2c946c788/cryptography-37.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cx-Oracle @ file:///data/zkci8im/.cache/pypoetry/artifacts/73/a1/48/01f42f211f534f73aed19b07156e2f58bc51f3e931b7ff24e916bc6183/cx_Oracle-8.3.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
cycler @ file:///data/zkci8im/.cache/pypoetry/artifacts/32/5f/b5/69e6bd574bcff38cf319bbc2937a6f71f47b5532fff18dfd7dc221591a/cycler-0.11.0-py3-none-any.whl
defer @ file:///data/zkci8im/.cache/pypoetry/artifacts/7f/d4/64/12aa60c68128ded2c3b83733ede1376668571bee98f2234f998d6fccfd/defer-1.0.4.tar.gz
defusedxml @ file:///data/zkci8im/.cache/pypoetry/artifacts/52/8e/a8/6b6077e99f00435c289f1e6d72413a518d187c42f4f16018e9748996aa/defusedxml-0.7.1-py2.py3-none-any.whl
Deprecated @ file:///data/zkci8im/.cache/pypoetry/artifacts/1b/df/01/0a69c6fa606522ad7535a9433d6f353e57e674a41f23b0c75851455ade/Deprecated-1.2.13-py2.py3-none-any.whl
diffsync @ file:///data/zkci8im/.cache/pypoetry/artifacts/25/f2/03/71d9fcbe3efd41c93400434afd2eecc791427d325371195d79b0ecba7e/diffsync-1.4.3-py3-none-any.whl
Django @ file:///data/zkci8im/.cache/pypoetry/artifacts/84/b8/62/2618b9e6130ebb2661834718637fd186dcaf9d4bbdbe5d8e40b5044e90/Django-3.1.14-py3-none-any.whl
django-ajax-tables @ file:///data/zkci8im/.cache/pypoetry/artifacts/41/4b/c2/0b226c181d25ebe682f1c331ff835fc913da1840e63970f43b2d4d5025/django_ajax_tables-1.1.1-py3-none-any.whl
django-appconf @ file:///data/zkci8im/.cache/pypoetry/artifacts/15/b0/42/820bd462dcd763df6043e873d416d949595b144ac76bda950a7059322e/django_appconf-1.0.5-py3-none-any.whl
django-cacheops @ file:///data/zkci8im/.cache/pypoetry/artifacts/77/ab/1a/858382fe25b8025eddd02dc56b2ed986f86d9343e51186984d4c87de0b/django_cacheops-5.1-py2.py3-none-any.whl
django-celery-beat @ file:///data/zkci8im/.cache/pypoetry/artifacts/3f/71/03/feac735288ca43683afd8f5e889f6c1670345f879acfb60f5391f4f559/django_celery_beat-2.2.1-py2.py3-none-any.whl
django-constance @ file:///data/zkci8im/.cache/pypoetry/artifacts/92/73/f6/b27afffb80673c8e7cff1f06405d06d8078cce3ccf8156b94349e7417e/django_constance-2.8.0-py3-none-any.whl
django-cors-headers @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/95/72/c2b1d92874644571f8e4b3b90951706ac3d96db0ae5f8ed50a196fc1b1/django_cors_headers-3.7.0-py3-none-any.whl
django-cryptography @ file:///data/zkci8im/.cache/pypoetry/artifacts/be/36/af/6730dd6bebebfddbcac7ffe452b9b6db03977659dad14b31f5417f1f85/django_cryptography-1.0-py3-none-any.whl
django-db-file-storage @ file:///data/zkci8im/.cache/pypoetry/artifacts/73/ed/f1/84bb26eaf28ee7772f62bf319f95bc47b58964c9c45758b73b7df111ee/django-db-file-storage-0.5.5.tar.gz
django-downloadview @ file:///data/zkci8im/.cache/pypoetry/artifacts/b1/2e/86/bbffe2f5c2568e7f427eb695e6b21f312104be6bbd968b66012ff5804a/django_downloadview-2.3.0-py3-none-any.whl
django-entangled @ file:///data/zkci8im/.cache/pypoetry/artifacts/2e/37/48/145379ef54ff01cb65af0eb2397a87e83fdf209dcc1aaebb1e4a913bbd/django_entangled-0.5.3-py3-none-any.whl
django-extensions @ file:///data/zkci8im/.cache/pypoetry/artifacts/e4/44/36/92d6e2883b730fa9f5870d82725511e42c87732a524094cef75e9080b9/django_extensions-3.1.5-py3-none-any.whl
django-filter @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/56/fc/d46f2e843498b83624f789ec451754d9caa1da8e04dafcbbb61d273a1e/django_filter-2.4.0-py3-none-any.whl
django-health-check @ file:///data/zkci8im/.cache/pypoetry/artifacts/68/fa/24/e2b9ae1477deafa6b5dcbf3878000c6eca08b90da6cbb2ef09b31cfe3a/django_health_check-3.16.5-py2.py3-none-any.whl
django-jinja @ file:///data/zkci8im/.cache/pypoetry/artifacts/7d/7d/e5/ebaf16f467f945ac40af64aa5470aa6c16b6d15d1bba0d941c128ddf3d/django_jinja-2.7.1-py3-none-any.whl
django-js-asset @ file:///data/zkci8im/.cache/pypoetry/artifacts/8d/c3/d7/d7ec51ee2330df3de9033568464f80b75a61399948db022bf6a2faf604/django_js_asset-2.0.0-py3-none-any.whl
django-mptt @ file:///data/zkci8im/.cache/pypoetry/artifacts/26/12/ef/a313335a7225e0b4cd0f2b6d6cfe0057523ce7b5fb803295fd2dc7b25a/django_mptt-0.11.0-py2.py3-none-any.whl
django-ordered-model @ file:///data/zkci8im/.cache/pypoetry/artifacts/ac/a1/1c/7deedf732708cd7a331a55900332d66b6377a30637f28de520f7da79d0/django_ordered_model-3.6-py3-none-any.whl
django-picklefield @ file:///data/zkci8im/.cache/pypoetry/artifacts/a9/e2/04/f4ab1ce6c3ba035cb9f3492562ff2772f6bcd522690498e141ea225844/django_picklefield-3.0.1-py3-none-any.whl
django-prometheus @ file:///data/zkci8im/.cache/pypoetry/artifacts/a8/08/89/a058c01936e8c1ff21a3c3696258faf6dd5c236ad1ca7ce74ec7a2744a/django_prometheus-2.1.0-py2.py3-none-any.whl
django-redis @ file:///data/zkci8im/.cache/pypoetry/artifacts/e6/f4/01/66c9ff3f089f1e3cd1a4363cf0931618d904c54676ace18cfdf94a3f83/django_redis-5.2.0-py3-none-any.whl
django-rq @ file:///data/zkci8im/.cache/pypoetry/artifacts/ce/18/a5/8ca3fd96dee56aa6a520a9fbd8305feb9fd8cb1eab77d0fff267e2d872/django_rq-2.4.1-py2.py3-none-any.whl
django-storages @ file:///data/zkci8im/.cache/pypoetry/artifacts/84/52/bb/5ac21f52e4e07dcddd223ea08d7ec643a9ae872226ad79b1e4daed3769/django_storages-1.12.3-py3-none-any.whl
django-tables2 @ file:///data/zkci8im/.cache/pypoetry/artifacts/78/46/84/69794e310d82527ae5e95fefc3fdde4d223a76d7e9dcbf58446b61e597/django_tables2-2.3.4-py2.py3-none-any.whl
django-taggit @ file:///data/zkci8im/.cache/pypoetry/artifacts/0a/ba/45/8fd1c16264accb6c7b76eb13fd536bdbc36bce8e5e4f37f9d976a834c0/django_taggit-1.3.0-py3-none-any.whl
django-timezone-field @ file:///data/zkci8im/.cache/pypoetry/artifacts/bc/1a/83/6e3f3fd42c1cb4bc4744298de8f3eee498573bd3a5d45e87b0dd58eb4b/django_timezone_field-4.1.2-py3-none-any.whl
django-webserver @ file:///data/zkci8im/.cache/pypoetry/artifacts/39/e2/f5/581bc4bba43156a12191374c64b3de7e357126ab5a32b51e8c832ad52b/django_webserver-1.2.0-py2.py3-none-any.whl
djangorestframework @ file:///data/zkci8im/.cache/pypoetry/artifacts/a0/f6/64/3c17dd26b71862f114e3edeb3fbbe5a113b97fbabfc02ddc01ac73d325/djangorestframework-3.12.4-py3-none-any.whl
drf-yasg @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/b1/f2/b6d572c8bf1b12ea52c42f019d95c72855926fbf2e4a21158eb68c9425/drf_yasg-1.20.0-py2.py3-none-any.whl
ecdsa @ file:///data/zkci8im/.cache/pypoetry/artifacts/88/fc/5e/0169779073adb3d1920c38c24f1b47601e447622ec221514fdd9cdb1cb/ecdsa-0.17.0-py2.py3-none-any.whl
et-xmlfile @ file:///data/zkci8im/.cache/pypoetry/artifacts/fd/6d/4f/3e758a754df0ad5b0be20ae9e4c3a4d3b07e778b38a0ff76c980cb6f9d/et_xmlfile-1.1.0-py3-none-any.whl
fonttools @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/40/9d/ad9a026cdfc88d8ab195b1228e254f69a91a3a7084d69b4ff79de5ba6d/fonttools-4.33.3-py3-none-any.whl
funcy @ file:///data/zkci8im/.cache/pypoetry/artifacts/ff/29/3c/7612d8bae9b3af86333d22fe6156fa13ae2d26c70803da765b7f87f8ed/funcy-1.17-py2.py3-none-any.whl
GES.Entitlements @ file:///data/zkci8im/.cache/pypoetry/artifacts/76/1d/60/7e657bbeee6cc1becde51c9d21ed5618547b8d71a5e42fe2e02c0c2275/GES.Entitlements-7.4.0.0-py3-none-any.whl
gitdb @ file:///data/zkci8im/.cache/pypoetry/artifacts/20/48/e1/e92559540498e6fc81acc2b79725aed4831869ac4930e4875f042a3bc6/gitdb-4.0.9-py3-none-any.whl
GitPython @ file:///data/zkci8im/.cache/pypoetry/artifacts/45/99/a3/8dc50fe3361368d219d1dc12474ce03ee847dede7e83853f36238f7525/GitPython-3.1.18-py3-none-any.whl
graphene @ file:///data/zkci8im/.cache/pypoetry/artifacts/e4/b4/e5/ebf568635788b1888e94de08f61bf6df7bc035c66fdb8130caccde0534/graphene-2.1.9-py2.py3-none-any.whl
graphene-django @ file:///data/zkci8im/.cache/pypoetry/artifacts/ed/bb/91/d0a9103f49ff61786d14c64d62b62d3bf3ac06aee2f44eed7f19c7d285/graphene_django-2.15.0-py2.py3-none-any.whl
graphene-django-optimizer @ file:///data/zkci8im/.cache/pypoetry/artifacts/e7/41/e6/a86f233609fe6e007737bc3960a4d869720e2d7f6fbbac49fd76a7b90f/graphene-django-optimizer-0.8.0.tar.gz
graphql-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/42/0a/5d/522bcaafc471de5533ac9aeef689edc9402c048bc5566cc746f7e17928/graphql_core-2.3.2-py2.py3-none-any.whl
graphql-relay @ file:///data/zkci8im/.cache/pypoetry/artifacts/2d/bb/de/db5baddb597438d931a9cb986596b9382ab4153c628523bd52e4446fd6/graphql_relay-2.0.1-py3-none-any.whl
ibm-db @ file:///data/zkci8im/.cache/pypoetry/artifacts/40/f3/30/bf498333533e3962d2dd380fa2c15dd56280b298d4d6d29e57373c2c4e/ibm_db-3.1.1.tar.gz
idna @ file:///data/zkci8im/.cache/pypoetry/artifacts/6f/d1/a9/d0625f343d7d25e3e0effced7539d67c94c0426cd32bdf4fbb3a7528f0/idna-3.3-py3-none-any.whl
importlib-metadata @ file:///data/zkci8im/.cache/pypoetry/artifacts/b6/26/b1/de9225713e00b48ca565477825c09a513cf6e597d5b32a6f11a74f1eb8/importlib_metadata-4.4.0-py3-none-any.whl
inflection @ file:///data/zkci8im/.cache/pypoetry/artifacts/98/bc/19/dd3ca5a6cc7c1d2ebc3fdeb01ae68dc6ec59c3a27f27d6f8ba2ed7560e/inflection-0.5.1-py2.py3-none-any.whl
isodate @ file:///data/zkci8im/.cache/pypoetry/artifacts/7f/4c/a9/6ee9a5be06dbc032cddbdd42142dc7828a67c6d959f9cc7eda6f6cec3a/isodate-0.6.1-py2.py3-none-any.whl
itypes @ file:///data/zkci8im/.cache/pypoetry/artifacts/27/5d/31/a7c551c3439cd7711a903bf67c5313d27a79bb908adcb734413f6ff3b8/itypes-1.2.0-py2.py3-none-any.whl
Jinja2 @ file:///data/zkci8im/.cache/pypoetry/artifacts/50/1b/94/55cae878ce19751af0ec5f40765ef51e5af01c903cad84e0cdeaa3d7f0/Jinja2-2.11.3-py2.py3-none-any.whl
jmespath @ file:///data/zkci8im/.cache/pypoetry/artifacts/65/c5/84/a16f584a42b02584726b43531ed19ca400f538f299a302ba4ad88b4cb3/jmespath-1.0.0-py3-none-any.whl
jsonschema @ file:///data/zkci8im/.cache/pypoetry/artifacts/d7/ea/c1/138bfd39b9e843940605bd7170380281f7e0df8fb2a47fa5d70974fe90/jsonschema-3.2.0-py2.py3-none-any.whl
kiwisolver @ file:///data/zkci8im/.cache/pypoetry/artifacts/90/ec/b4/941f119eada2e8221280a7b23e75a5d3e24ca089327c4cdca627412e85/kiwisolver-1.4.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
kombu @ file:///data/zkci8im/.cache/pypoetry/artifacts/b0/16/df/66aefa2d180ceace98dd7c2b0b1014c4380669e7382214e953aeed23ab/kombu-5.2.4-py3-none-any.whl
lxml @ file:///data/zkci8im/.cache/pypoetry/artifacts/35/b0/67/93757e946c5baba990e7c2c1e16f618c9073f09d6ae033d506db95ce5a/lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
Markdown @ file:///data/zkci8im/.cache/pypoetry/artifacts/34/32/83/707e90466826044af311cf500aca2bc915a5bbecd661201fbce6a7cf77/Markdown-3.3.7-py3-none-any.whl
MarkupSafe @ file:///data/zkci8im/.cache/pypoetry/artifacts/18/2e/dc/4e3a84074e8f7e62f0badbe8fd7e1c027f256893e82d756e7153c758d1/MarkupSafe-2.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
matplotlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/0f/f5/b5/8d4616bd0e92bad21257d238a1cb30c3a643277a1eb7887e274b140ccf/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
mysqlclient @ file:///data/zkci8im/.cache/pypoetry/artifacts/3e/49/38/62c5c2ca4417ce1aab974d62c8ed023b4731edb2ec0cf30e8700a817d2/mysqlclient-2.1.0.tar.gz
nautobot @ file:///data/zkci8im/.cache/pypoetry/artifacts/a6/02/d4/096e1c7aade77e4b2ec96f51bcafd5d384ee2a705847dde9423a24bf8e/nautobot-1.2.11-py3-none-any.whl
nautobot-anap-circuits @ file:///data/zkci8im/.cache/pypoetry/artifacts/63/b5/0d/e8252d47044b6425c000fe541a92a2562ecd22d9a0f24a790fb8b934a4/nautobot_anap_circuits-0.3.15-py3-none-any.whl
nautobot-anap-circuits-sync @ file:///data/zkci8im/.cache/pypoetry/artifacts/4c/c5/58/a610a0576329c19eedcad609c2c4ffd9461819b1e73b80a982eb0ff53c/nautobot_anap_circuits_sync-0.3.6-py3-none-any.whl
nautobot-anap-symphony @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/39/86/f78b31665c81bdddf246d3272a5907d4996a6c861c80f92e9f1dc3455a/nautobot_anap_symphony-0.6.44-py3-none-any.whl
nautobot-bofa-ipam @ file:///data/zkci8im/.cache/pypoetry/artifacts/81/a8/a2/96f2ac7d2200b988b7a731f8e45efef6c3263fb7b823175e9d0e578e29/nautobot_bofa_ipam-1.4.3-py3-none-any.whl
nautobot-bpa @ file:///data/zkci8im/.cache/pypoetry/artifacts/22/d0/80/7de73e8e798b5c9f6a99223f59db1ec60dbaf6da104c56769a014d3e20/nautobot_bpa-2.1.0-py3-none-any.whl
nautobot-cert-letters @ file:///data/zkci8im/.cache/pypoetry/artifacts/ff/a5/0a/2e65e3eebb1f67727efffa57e9616d043e2dd7501d2e432b10fa8ae471/nautobot_cert_letters-0.1.17-py3-none-any.whl
nautobot-clearpass @ file:///data/zkci8im/.cache/pypoetry/artifacts/02/d7/8b/84e2b6ff52cf8a23bee6f935d8cdc881668f43c25afa704d90735894e1/nautobot_clearpass-1.0.9-py3-none-any.whl
nautobot-cmdb @ file:///data/zkci8im/.cache/pypoetry/artifacts/d4/90/8a/4e5bbf6bf6b32de2d33e3ba2330ea72cf19e34ee6abaf2fc30a8e4a81d/nautobot_cmdb-1.13.5-py3-none-any.whl
nautobot-common @ file:///data/zkci8im/.cache/pypoetry/artifacts/b5/de/92/7de8b566158ff1df03b33c41f8c87e84eabb8c27fe09d0a94e47e392f3/nautobot_common-0.3.5-py3-none-any.whl
nautobot-cyberark @ file:///data/zkci8im/.cache/pypoetry/artifacts/48/e4/68/148f4623324c647d1733b23fa41b93a906ca17ad4f681d177528773617/nautobot_cyberark-2.2.0-py3-none-any.whl
nautobot-data-classification @ file:///data/zkci8im/.cache/pypoetry/artifacts/25/90/36/a34aeabb27539900ee5341cd73aae55f85ee6ac6cb4e43355645965ec2/nautobot_data_classification-1.0.1-py3-none-any.whl
nautobot-data-ownership @ file:///data/zkci8im/.cache/pypoetry/artifacts/52/37/8f/72de92d33a4ff9ac987f52427afa3d7ae7ed222ac34b06885d0c2361c0/nautobot_data_ownership-0.2.0-py3-none-any.whl
nautobot-device-lifecycle-mgmt @ file:///data/zkci8im/.cache/pypoetry/artifacts/d2/c8/5f/cda93c10d2f60155cc1a5fb3e359d291ef2ebf69903032b3524d7cbbbc/nautobot_device_lifecycle_mgmt-0.4.1-py3-none-any.whl
nautobot-eps @ file:///data/zkci8im/.cache/pypoetry/artifacts/8d/86/03/b34e76b9b4c08aca090def035abbb5ec5d56fc30a710d3934c6eb1c2b5/nautobot_eps-0.1.0-py3-none-any.whl
nautobot-network-classification @ file:///data/zkci8im/.cache/pypoetry/artifacts/c8/a6/2a/a00e72efb95bac27c3903d1f04ac888b19383e16c312f447952e67c134/nautobot_network_classification-1.1.0-py3-none-any.whl
nautobot-nidm-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/f8/e1/e03db47852cad4fd24b84c92d6dda32a5d3e68448a66cd23c0eadc5c7d/nautobot_nidm_core-1.2.1-py3-none-any.whl
nautobot-rdr @ file:///data/zkci8im/.cache/pypoetry/artifacts/b9/d0/bb/db1717b5380cc3bb98c70bde3190dd278194e350fc7e1b3ebcbb68492b/nautobot_rdr-1.0.1-py3-none-any.whl
nautobot-sevone-onboarding @ file:///data/zkci8im/.cache/pypoetry/artifacts/5a/b0/ee/48d5c3db983b7c208d26b32de54965f275a0408f507dd87294408b4b97/nautobot_sevone_onboarding-0.0.1-py3-none-any.whl
nautobot-ssot @ file:///data/zkci8im/.cache/pypoetry/artifacts/15/19/8a/6aa9f5ad8232fa1c9e196fed6bd113b0d4f7cab6fd67d4a4c50f6a0c94/nautobot_ssot-1.1.2-py3-none-any.whl
nautobot-standards @ file:///data/zkci8im/.cache/pypoetry/artifacts/e3/23/18/61dcab40bb5e3561a0cb412be3917595c317131a9211f19c575cf4ed24/nautobot_standards-1.8.5-py3-none-any.whl
netaddr @ file:///data/zkci8im/.cache/pypoetry/artifacts/f9/16/e8/d9224c3975d478de3009d1d47712cf111b0a4b5d854dc9e9fabc65dbde/netaddr-0.8.0-py2.py3-none-any.whl
netutils @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/db/16/4111ea796f3afb67d33e86f63f35aba5a21019cda3f95e903ba63e8762/netutils-1.0.0-py3-none-any.whl
numpy @ file:///data/zkci8im/.cache/pypoetry/artifacts/de/d5/28/b0cfd2d4c3e9a1c8ae4a78490049d8a3412d697a7f0a5b34c44f4d1d2b/numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
oauthlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/07/0e/03/4277e4ccf4acbe4361be783f58dd746cb95b78649a558edf1307a5a1f6/oauthlib-3.2.0-py3-none-any.whl
openpyxl @ file:///data/zkci8im/.cache/pypoetry/artifacts/e8/b7/32/e56fe150a5abcd4739833b3a0a1548d230679b29074ef42402272629c9/openpyxl-3.0.10-py2.py3-none-any.whl
packaging @ file:///data/zkci8im/.cache/pypoetry/artifacts/3b/2b/d0/229eacac699b7990b07c4a20d0f449f10465f21747f61e1b16ca0788b6/packaging-21.3-py3-none-any.whl
Pillow @ file:///data/zkci8im/.cache/pypoetry/artifacts/47/f1/78/5bbff537c3d88fb8237793032585a247723b14e00240af6cb68de37365/Pillow-9.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
platformdirs @ file:///data/zkci8im/.cache/pypoetry/artifacts/45/0a/73/b60f9509a6bc1223523fb621fa706e105b8b75953ba5f6b20671b297ce/platformdirs-2.5.2-py3-none-any.whl
prometheus-client @ file:///data/zkci8im/.cache/pypoetry/artifacts/29/46/d1/ae1047fbfd3702e1bcdc496b5ac6058f6522256abda4f7da8893f32891/prometheus_client-0.14.1-py3-none-any.whl
promise @ file:///data/zkci8im/.cache/pypoetry/artifacts/62/2a/40/f263aa397064cb03a099f84abc98a7f75bdc1a34962f4eb0012ea78a53/promise-2.3.tar.gz
prompt-toolkit @ file:///data/zkci8im/.cache/pypoetry/artifacts/a6/f6/75/fb1c4bae58f6c4622619dc9dbb1733b76f7ff3c0578d91c15ae91feae7/prompt_toolkit-3.0.29-py3-none-any.whl
psycopg2-binary @ file:///data/zkci8im/.cache/pypoetry/artifacts/96/8d/16/3d21fd6c74323042ba23e37cbdb4d0d2a2bcfaa8e50d5e394378ca4a38/psycopg2_binary-2.8.6-cp38-cp38-manylinux1_x86_64.whl
pyasn1 @ file:///data/zkci8im/.cache/pypoetry/artifacts/44/6a/a9/c21cf90c1e0756f209e70aa7f38486615fed1dd0d0aea32e06f20ddb4c/pyasn1-0.4.8-py2.py3-none-any.whl
pycountry @ file:///data/zkci8im/.cache/pypoetry/artifacts/42/ef/f9/4f6319d77b9ae31577947642412faf82f7bfe8bc6a9bd52ea490088cba/pycountry-20.7.3.tar.gz
pycparser @ file:///data/zkci8im/.cache/pypoetry/artifacts/33/0c/a0/326b0a2cf7e136e67ea4dbdbcfe7147a7db9c8eac3f45eaf5120b95303/pycparser-2.21-py2.py3-none-any.whl
pycryptodome @ file:///data/zkci8im/.cache/pypoetry/artifacts/5a/a7/db/1f6483e9468f56bf8129100ace0c108433c24ebb8302620d1db5caa52c/pycryptodome-3.10.4-cp35-abi3-manylinux2010_x86_64.whl
pydantic @ file:///data/zkci8im/.cache/pypoetry/artifacts/6b/fd/fb/a8b33721ecebb58ba4b2fde8b2e799fb03ef090283041c8ff8136cb96c/pydantic-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
PyJWT @ file:///data/zkci8im/.cache/pypoetry/artifacts/4e/58/c4/35a0ce0b263912f8cb70102a00ea6ad3e4df070fc1999acd502193b950/PyJWT-2.4.0-py3-none-any.whl
pymssql @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/f4/f0/5a68c58d9ad2ca40ecbedd7675d11f7ecc10d2efebb0f7ad38b2d52c4f/pymssql-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pyparsing @ file:///data/zkci8im/.cache/pypoetry/artifacts/fc/60/07/1b440c7539c1662fbae8276892b463f7ff5daa3dbde20a91709721264d/pyparsing-3.0.9-py3-none-any.whl
pyrsistent @ file:///data/zkci8im/.cache/pypoetry/artifacts/b3/07/67/b2f8ed0cbef09958ed5340a50cfb1649662be07ba14110e999b6cfda0c/pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
python-crontab @ file:///data/zkci8im/.cache/pypoetry/artifacts/80/4f/88/386611efe39c69c86592f1eb8d3c44f6dc730192df536ca8351a5e7721/python-crontab-2.6.0.tar.gz
python-dateutil @ file:///data/zkci8im/.cache/pypoetry/artifacts/a8/e2/44/bdcaa1ff9495f3ad9e18dd6e63bb9e887647aba953d9ad9fc05a86fae7/python_dateutil-2.8.2-py2.py3-none-any.whl
python-jose @ file:///data/zkci8im/.cache/pypoetry/artifacts/f6/2a/b8/b6494aa866f7cc75ab568808374a5b083ffdc16206e8f3b289ee18c2b0/python_jose-3.3.0-py2.py3-none-any.whl
python3-openid @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/12/9f/ea7f74cf5238a45aadf81b3b7eadcfefa8951666c12b69fec11aa464bd/python3_openid-3.2.0-py3-none-any.whl
python3-saml @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/dc/e4/ebc0a921e5a1728077b6deb835157f90121645d5052b614e3982f3d670/python3_saml-1.12.0-py3-none-any.whl
pytz @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/34/d6/c45c9e208f555808c5cd019789091f10dd40a2f3022649a3f20800d62a/pytz-2022.1-py2.py3-none-any.whl
pyuwsgi @ file:///data/zkci8im/.cache/pypoetry/artifacts/94/5a/2a/3ebdbc0013fd6046b29a1b273279cdbe5092fbf951c00a490d69f83c42/pyuwsgi-2.0.20-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
PyYAML @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/3b/da/8e567f27818a41bf1549e53b5cd23647b1d7e2e647e0822dabb01a63bb/PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl
redis @ file:///data/zkci8im/.cache/pypoetry/artifacts/af/f2/d3/bd5e4c2a389a010eee4acc4f08add48552e1be122bc7c768db5b8375ab/redis-4.3.1-py3-none-any.whl
requests @ file:///data/zkci8im/.cache/pypoetry/artifacts/02/17/db/7a356bc4171d71bd271a00d6df46c824f4370f2ef39d88b74e6083bc03/requests-2.27.1-py2.py3-none-any.whl
requests-file @ file:///data/zkci8im/.cache/pypoetry/artifacts/82/c0/ba/0889eaaa9e09451feea27723a96f52056ef402da06ae9e2b0b785d1879/requests_file-1.5.1-py2.py3-none-any.whl
requests-oauthlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/b7/88/21/a294759fa4bda84604868791a9fb7ae94d995a0a5a3e446867437b62ef/requests_oauthlib-1.3.1-py2.py3-none-any.whl
requests-toolbelt @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/d0/4e/12d5b1fabbdf8cc813b65a85b4a44885ddccf80ddbe42cdab87df3c9c4/requests_toolbelt-0.9.1-py2.py3-none-any.whl
resolvelib @ file:///data/zkci8im/.cache/pypoetry/artifacts/95/f0/cd/33d25064e3e20dd4c1ad2fa315e04cff25fd0c6d6ad79b9068a3c33af2/resolvelib-0.5.5-py2.py3-none-any.whl
robot-bpa-sdk @ file:///data/zkci8im/.cache/pypoetry/artifacts/99/87/39/0a25e697d3f52ae3c0b2520d8249d5abb5573d30222ec78f51518d79e4/robot_bpa_sdk-2021.7.15-py3-none-any.whl
rq @ file:///data/zkci8im/.cache/pypoetry/artifacts/d6/8c/a8/a5173f44dfe1a74a0593828ff56e0245df984f348daadd3560c5c1024c/rq-1.10.1-py2.py3-none-any.whl
rsa @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/83/7d/515ad00d6559117fd24ee2dd69dbfdaa556c8986e6e35879cabad091d2/rsa-4.8-py3-none-any.whl
ruamel.yaml @ file:///data/zkci8im/.cache/pypoetry/artifacts/d6/a3/d5/50b355606c7e8cbf1d3e88a8da1b29c1957319ceb41a587c45aca1ee5c/ruamel.yaml-0.17.21-py3-none-any.whl
ruamel.yaml.clib @ file:///data/zkci8im/.cache/pypoetry/artifacts/da/9f/c3/0b970a7488666902b22784c2305a321e4fc5176d96eee119d8ea9c053f/ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl
Rx @ file:///data/zkci8im/.cache/pypoetry/artifacts/34/ec/6f/bc1186029e7ad679cf82dde5131f5790bc62d83b3ba2aa0500222de304/Rx-1.6.1-py2.py3-none-any.whl
s3transfer @ file:///data/zkci8im/.cache/pypoetry/artifacts/20/db/c0/28a2baf88f972584bd05f1495d1a7a4c4e0a318b9831c56534a13df0b5/s3transfer-0.5.2-py3-none-any.whl
setuptools-scm @ file:///data/zkci8im/.cache/pypoetry/artifacts/c1/0e/d6/bf340b99a2ade5d92443240719f8a443e7bd88b591bcef74956a16899a/setuptools_scm-6.4.2-py3-none-any.whl
singledispatch @ file:///data/zkci8im/.cache/pypoetry/artifacts/49/64/ca/e60267abe7b859719ae4e08a0422e8f56ad4a98b8767fb76a8272deb99/singledispatch-3.7.0-py2.py3-none-any.whl
six @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/55/4d/44b471ed2a50fcce09e4bda1a06c8a7ec642c53b5166ce5c0097275802/six-1.16.0-py2.py3-none-any.whl
smmap @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/bf/ea/1042000312501eb13606330aaa12de6e8c3a57ed7d33f1d659eef2e720/smmap-5.0.0-py3-none-any.whl
social-auth-app-django @ file:///data/zkci8im/.cache/pypoetry/artifacts/66/15/43/2212ab82e7ada52755f3131966a17d91a60bdd662aab49f5b1beb5b559/social_auth_app_django-4.0.0-py3-none-any.whl
social-auth-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/c7/fe/5b/dd39fab35925f9c3c27852572b608f78b47335bd33238cd16a47a12110/social_auth_core-4.1.0-py3-none-any.whl
sqlparse @ file:///data/zkci8im/.cache/pypoetry/artifacts/35/2a/c7/b94bd2500225f7c258ebddb92714290de3566d4a03a5f849b4c2b47c7c/sqlparse-0.4.2-py3-none-any.whl
structlog @ file:///data/zkci8im/.cache/pypoetry/artifacts/1d/1c/fd/7df0160dbd1830fdb46c740b0a81ea252fb6a73b0677353ebaa3f0d674/structlog-20.2.0-py2.py3-none-any.whl
svgwrite @ file:///data/zkci8im/.cache/pypoetry/artifacts/2e/b0/bb/551a83c1df6d04ead4a3e7370f68d0b57e02fad656042cad5df2a310ed/svgwrite-1.4.2-py3-none-any.whl
swagger-spec-validator @ file:///data/zkci8im/.cache/pypoetry/artifacts/71/71/63/7a52697831713b439971a032fb83e31dd1df44aa638e90be82a5ac3d5d/swagger_spec_validator-2.7.4-py2.py3-none-any.whl
text-unidecode @ file:///data/zkci8im/.cache/pypoetry/artifacts/38/cc/09/814f7784e36c8140d07513c0e1709fd9a3807cdd27e6be6efbbdb628fa/text_unidecode-1.3-py2.py3-none-any.whl
tomli @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/1c/82/1fc4d637d7cad4a6d39e21ab22bd7957ec3228f36c499c4b2d6ff33b4e/tomli-2.0.1-py3-none-any.whl
types-PyYAML @ file:///data/zkci8im/.cache/pypoetry/artifacts/85/94/c2/81254c1bdbfd106a966adca8c00f5cb3a20cdcee1cd50975f3fa345019/types_PyYAML-6.0.7-py3-none-any.whl
typing_extensions @ file:///data/zkci8im/.cache/pypoetry/artifacts/0d/6f/04/b8bd0c34c092d479e1f9ee7ac241d6f264cc01db2832e08fd4a8c1fb3e/typing_extensions-4.2.0-py3-none-any.whl
uritemplate @ file:///data/zkci8im/.cache/pypoetry/artifacts/76/08/ec/048fc3fa6c2480f858b637ddfe52729642b79c01e8f47ce663d74df8d7/uritemplate-4.1.1-py2.py3-none-any.whl
urllib3 @ file:///data/zkci8im/.cache/pypoetry/artifacts/69/80/ba/5408bb5fec831682384684c11eab54da6b8dd997fb85871eba696966b2/urllib3-1.26.9-py2.py3-none-any.whl
vine @ file:///data/zkci8im/.cache/pypoetry/artifacts/d0/af/cd/40d4b255297e753565b27ec2cecb352311be0f65fb821c71a02d9ed109/vine-5.0.0-py2.py3-none-any.whl
wcwidth @ file:///data/zkci8im/.cache/pypoetry/artifacts/16/12/ba/6211db7a12b3c00978b51a85563bdb0d710e9397027fa8f9c8025daa76/wcwidth-0.2.5-py2.py3-none-any.whl
wrapt @ file:///data/zkci8im/.cache/pypoetry/artifacts/9b/83/b3/ee7bc92d3d7376c36e018277e1cd90f052e70ce5d76bf5c672ec271d34/wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
xmlsec @ file:///data/zkci8im/.cache/pypoetry/artifacts/d4/76/2c/871220815f373e94acaa810926b0e857a351d3b24e87a258dc72a2e0c1/xmlsec-1.3.12.tar.gz
zeep @ file:///data/zkci8im/.cache/pypoetry/artifacts/3a/55/89/f82f1d16b4e53c6a318a21628bb26e6921baf442bd1cf65eeeb3fe6920/zeep-4.1.0-py2.py3-none-any.whl
zipp @ file:///data/zkci8im/.cache/pypoetry/artifacts/6d/cc/4e/46283fca4ce6eff04a6e6bfe21e67d2c54e2e6bc27ba1d60c7be3bc0da/zipp-3.8.0-py3-none-any.whl

Other Dependencies

Python: 3.8 celery: 5.2.7 django_celery_beat: 2.2.1

Minimally Reproducible Test Case

Expected Behavior

A single task (both one-off, and periodic) - should execute ONCE AND ONLY ONCE .

Actual Behavior

When multiple tasks are started at roughly the same time, and one task completes while other task(s) still running - running task(s) get started again resulting in MULTIPLE TASK EXECUTIONS.

Package Versions

celery: 5.2.7
django_celery_beat: 2.2.1

Issue Description

Celery beat Scheduler (re)starts same task while task is running, resulting in multiple executions of the same task.

Details

Consider celery.beat.Scheduler.tick() implementation (code below is from v5.2.7):

# pylint disable=redefined-outer-name
def tick(self, event_t=event_t, min=min, heappop=heapq.heappop,
             heappush=heapq.heappush):
        """Run a tick - one iteration of the scheduler.

        Executes one due task per call.

        Returns:
            float: preferred delay in seconds for next call.
        """
        adjust = self.adjust
        max_interval = self.max_interval

        if (self._heap is None or
                not self.schedules_equal(self.old_schedulers, self.schedule)):
            self.old_schedulers = copy.copy(self.schedule)
            self.populate_heap()

        H = self._heap

        if not H:
            return max_interval

        event = H[0]
        entry = event[2]
        is_due, next_time_to_run = self.is_due(entry)
        if is_due:
            verify = heappop(H)
            if verify is event:
                next_entry = self.reserve(entry)
                self.apply_entry(entry, producer=self.producer)
                heappush(H, event_t(self._when(next_entry, next_time_to_run),
                                    event[1], next_entry))
                return 0
            else:
                heappush(H, verify)
                return min(verify[0], max_interval)
        return min(adjust(next_time_to_run) or max_interval, max_interval)

In this implementation, self.is_due() is executed and then (if is_due == True is returned) - self.apply_entry() sends event to start task execution to the celery worker (assuming 'verify' is event):

is_due, next_time_to_run = self.is_due(entry)

Now, consider case where given task is already executing... Since task is executing, it is already "due" - therefore, self.is_due() returns True (is_due == True). Scheduler.tick() happily proceeds and executes self.apply_entry() - which essentially starts same task's execution (by the worker) again. This continues on every Scheduler.tick() execution - until self.is_due() returns False - resulting in same task (single scheduled instance of the task) executed multiple times.

Problem summary

Celery beat Scheduler does not "recognize" that given task is already executing (running) - thus, sends event to start same task over and over again.

Additional details

I work on a project where multiple one-off tasks can be executed at any given moment. We have noticed (and proved that to be true) that same task is often executed multiple times (while some - only once). Having same task executed more than once - "kills" this project; it causes many negative side effects that cannot be handled or predicted (at least, not all of them).

We spent a lot of time troubleshooting this issue - which included debugging (use of breakpoints and stepping through) underlying Celery implementation. This troubleshooting led us to understanding the root cause of the issue: Celery does not recognize that task is "running" and solely relies on is_due() response to send event to start task execution.

We have also proved that same case applies not only to one-off tasks, but also to periodic scheduled tasks: periodic task that should run let's say once a day, or once an hour - can get executed multiple times during same day (or same hour) while initial task is running.

Conclusion

This is a HUGE issue (at least for us). Single task (single "instance" of a task) - must be executed ONCE AND ONLY ONCE. Celery scheduler's inability to recognize that task is already executing and prevent subsequent executions - is a VERY SUBSTANTIAL BUG, causing lots of unpredictable side effects.

To resolve this issue, some sort of "task is running" recognition must be implemented (either within Scheduler.tick() or (more likely) within is_due() method - that recognizes that task is already running and prevents subsequent task executions (while it is actually "is due").

NOTICE: Django celery beat is used, so particular is_due() method that gets executed - is: django_celery_beat.schedulers.ModelEntry.is_due().

Steps Taken

Following steps have been taken while trying to eliminate this issue:

• CELERY_ENABLE_UTC config. parameter set to True - to eliminate possible TZ related schedule discrepancies (this suggestion is reported in many discussion threads)
• --task-events command line argument is added to 'celery worker' command - to ensure that worker reports task events
• An attempt was made to set CELERYBEAT_SYNC_EVERY to 1 - to ensure tick() is executed after each task's completion

Steps to reproduce

1. Schedule two (or more) tasks to start at roughly the same time
2. When one task completes, scheduler sync (tick()) is executed
3. If the other task is still running, is_due() returns True - resulting in tick() execute self.apply_entry() which essentially starts same task (the one that is already executing) to execute again

[open-collective-bot]

Hey @vpogrebi 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[vpogrebi]

Sample celery beat log (portion) for demonstration:

[2022-06-07 17:04:11,297: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2022-06-07 17:04:11,602: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-enable-2022-06-07-17:04:09.182051_b70fd6d8-4f3a-4d8d-8069-0b0d09ddc4f9 (nautobot.extras.jobs.scheduled_job_handler)
[2022-06-07 17:04:11,785: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57 (nautobot.extras.jobs.scheduled_job_handler)
[2022-06-07 17:04:17,094: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2022-06-07 17:04:17,490: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57 (nautobot.extras.jobs.scheduled_job_handler)

In this log - observe following:

1. Two tasks were started at the same time ("Scheduler: Sending due task" logs at 2022-06-07 17:04:11)
2. Few seconds later, one task completed - indicated by the "DatabaseScheduler: Schedule changed" log (2022-06-07 17:04:17,094)
3. This schedule change event triggered scheduler sync - which in turn executed tick() call with event argument corresponding to the second (still running) task.
4. In tick(), self.is_due(entry) returned True (task is due) - which resulted in scheduler send another message to start this second task: "Scheduler: Sending due task" log at 2022-06-07 17:04:17,490

End result - one of the tasks (AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57) executed twice.

Additional thoughts

I noticed that every time this happens - two (or more) tasks are being kicked off by the same (single) sync event:

• Single "DatabaseScheduler: Schedule changed" log followed by ...
• Multiple (two or more) "Scheduler: Sending due task" logs

I am not sure if this fact has anything to do with this issue (bug) - but somehow I have strong feeling that it does. I suspect that these multiple tasks get "associated" with the same (single) event, and when one of the tasks completes - sync (tick()) evaluates each of the other tasks (schedule entries), discovers that they are "due" (is_due() returns True) and sends message to (re)start them again.

I may be wrong, but... this is one suspicion that I've got after giving this issue some more thought.

[auvipy]

you could just modify the previous issue. btw can you please try & check new release https://github.com/celery/django-celery-beat/releases/tag/v2.3.0 ?

[vpogrebi]

@auvipy

you could just modify the previous issue. btw can you please try & check new release https://github.com/celery/django-celery-beat/releases/tag/v2.3.0 ?

Turns out someone else from the team has upgraded our dev lane to django_celery_beat v2.3.0 - and that did not help.

[vpogrebi]

One other thought... django_celery_beat.schedulers.ModelEntry.is_due() - has this logic:

        # ONE OFF TASK: Disable one off tasks after they've ran once
        if self.model.one_off and self.model.enabled \
                and self.model.total_run_count > 0:
            self.model.enabled = False
            self.model.total_run_count = 0  # Reset
            self.model.no_changes = False  # Mark the model entry as changed
            self.model.save()
            # Don't recheck
            return schedules.schedstate(False, NEVER_CHECK_TIMEOUT)

When testing, I noticed that self.total_run_count does not match self.model.total_run_count when task (that is already running) is evaluated again - which results in starting same task again. I was able to test my suspicion by providing a custom override of celery.beat.Scheduler.apply_async() that offsets (corrects) this situation:

    def apply_async(self, entry, producer=None, advance=True, **kwargs):
        """Send event to the worker to start task execution.
       
        This is an override of the celery.beat.Scheduler.apply_async() method.
        After executing original apply_async() call, it synchronizes 'total_run_count'
        and saves model. This prevents same task from being started again while it is running.
        """
        resp = super().apply_async(entry, producer=None, advance=True, **kwargs)
        if entry.total_run_count != entry.model.total_run_count:
            entry.total_run_count = entry.model.total_run_count
            entry.model.save()
        return resp

Using this custom override - resolves this issue!

I was able to prove that by ensuring that self.total_run_count and self.model.total_run_count are same at a time apply_async() is executed - somehow prevents multiple executions of the same task. Therefore, I suspect that ensuring that these two versions of total_run_count match at a time apply_async() is executed - somehow ensures that self.model.total_run_count > 0 within django_celery_beat.schedulers.ModelEntry.is_due() which prevents duplicate task execution.

[auvipy]

I think we can move this issue to django-celery-beat now :)

[vpogrebi]

There is one other issue that is being resolved by the above "override" (listed in a comment above)... Before (the override), total_run_count stored in the database after job's execution - was unreliable: for the job that ran and completed - total_run_count could be 0, 1, 2, etc. For the job that executed once - it was either 0 or 1, and I could not figure out the "pattern", or predict in advance (while the job was running) whether it would end up with 0 or 1.

Somehow, my override also fixes that issue, too - all total_run_count are 1 for the one-off jobs.

[ja-gooding]

Are there any plans to fix these issues in any reasonable timeframe? Months or less? As far as I can tell, celery, especially celery beat, is fundamentally broken. It is not even able to know if a task is already running or not, and this causes issue, upon issue, upon issue, down the road.

There have been comments about this going back half a decade, with no root solution in sight. These issues and core fundamentals should have been thoughtfully and robustly tested with units-testing from day one with sound software engineering.

And please don't say if you have an issue; fork it or put in a pull request. At a certain point, the problems are too numerous and deep to fix. That's why I am asking if there is a plan to resolve these issues soon, or should we cut our losses and move on?

[pythobot]

I just upgraded to django-celery-beat==2.4.0 with celery==5.2.3, and that didn't help. now tasks duplicate even more.

Does anybody know how to solve this issue?

[mishra-ankit-dev]

Hi Team,

Is it still unresolved ? I am still facing the above stated issue.

I think the hotfix has been applied to apply_async, delay still has the same issue. apply_async seems to work fine with this fix added.