This document describes an older version of Celery (2.2). For the latest stable version please go here.

Internals: The worker


The worker consists of 4 main components: the consumer, the scheduler, the mediator and the task pool. All these components runs in parallel working with two data structures: the ready queue and the ETA schedule.


Data structures


The ready queue is either an instance of Queue.Queue, or celery.buckets.TaskBucket. The latter if rate limiting is enabled.


The ETA schedule is a heap queue sorted by time.



Receives messages from the broker using Kombu.

When a message is received it’s converted into a celery.worker.job.TaskRequest object.

Tasks with an ETA are entered into the eta_schedule, messages that can be immediately processed are moved directly to the ready_queue.


The schedule controller is running the eta_schedule. If the scheduled tasks eta has passed it is moved to the ready_queue, otherwise the thread sleeps until the eta is met (remember that the schedule is sorted by time).


The mediator simply moves tasks in the ready_queue over to the task pool for execution using celery.worker.job.TaskRequest.execute_using_pool().


This is a slightly modified multiprocessing.Pool. It mostly works the same way, except it makes sure all of the workers are running at all times. If a worker is missing, it replaces it with a new one.

Previous topic

Celery Deprecation Timeline

Next topic

Task Message Protocol

This Page