This document describes the current stable version of Celery (5.6). For development docs, go here.
Concurrency with gevent¶
Introduction¶
The gevent homepage describes it a coroutine -based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev or libuv event loop.
Features include:
Lightweight execution units based on greenlets.
API that reuses concepts from the Python standard library (for examples there are events and queues).
Cooperative DNS queries performed through a threadpool, dnspython, or c-ares.
Monkey patching utility to get 3rd party modules to become cooperative
TCP/UDP/HTTP servers
Subprocess support (through gevent.subprocess)
Thread pools
gevent is inspired by eventlet but features a more consistent API, simpler implementation and better performance. Read why others use gevent and check out the list of the open source projects based on gevent.
Enabling gevent¶
You can enable the gevent pool by using the
celery worker -P gevent or celery worker --pool=gevent
worker option.
$ celery -A proj worker -P gevent -c 1000
Note
When using the gevent pool via the Celery CLI (-P gevent/--pool=gevent),
Celery applies gevent monkey-patching early during startup.
If you need to monkey-patch manually (e.g., when embedding a worker), do it at the very start of the process
(before importing task modules or standard library I/O modules):
from gevent import monkey
monkey.patch_all()
Examples¶
See the gevent examples directory in the Celery distribution for some examples taking use of Eventlet support.
Known issues¶
There is a known issue using python 3.11 and gevent. The issue is documented here and addressed in a gevent issue. Upgrading to greenlet 3.0 solves it.
