This document describes the current stable version of Celery (3.1). For development docs, go here.
Periodic Tasks¶
Introduction¶
celery beat is a scheduler. It kicks off tasks at regular intervals, which are then executed by the worker nodes available in the cluster.
By default the entries are taken from the CELERYBEAT_SCHEDULE
setting,
but custom stores can also be used, like storing the entries
in an SQL database.
You have to ensure only a single scheduler is running for a schedule at a time, otherwise you would end up with duplicate tasks. Using a centralized approach means the schedule does not have to be synchronized, and the service can operate without using locks.
Time Zones¶
The periodic task schedules uses the UTC time zone by default,
but you can change the time zone used using the CELERY_TIMEZONE
setting.
An example time zone could be Europe/London:
CELERY_TIMEZONE = 'Europe/London'
This setting must be added to your app, either by configuration it directly
using (app.conf.CELERY_TIMEZONE = 'Europe/London'
), or by adding
it to your configuration module if you have set one up using
app.config_from_object
. See Configuration for
more information about configuration options.
The default scheduler (storing the schedule in the celerybeat-schedule
file) will automatically detect that the time zone has changed, and so will
reset the schedule itself, but other schedulers may not be so smart (e.g. the
Django database scheduler, see below) and in that case you will have to reset the
schedule manually.
Django Users
Celery recommends and is compatible with the new USE_TZ
setting introduced
in Django 1.4.
For Django users the time zone specified in the TIME_ZONE
setting
will be used, or you can specify a custom time zone for Celery alone
by using the CELERY_TIMEZONE
setting.
The database scheduler will not reset when timezone related settings change, so you must do this manually:
$ python manage.py shell
>>> from djcelery.models import PeriodicTask
>>> PeriodicTask.objects.update(last_run_at=None)
Entries¶
To schedule a task periodically you have to add an entry to the
CELERYBEAT_SCHEDULE
setting.
Example: Run the tasks.add task every 30 seconds.
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
'add-every-30-seconds': {
'task': 'tasks.add',
'schedule': timedelta(seconds=30),
'args': (16, 16)
},
}
CELERY_TIMEZONE = 'UTC'
Note
If you are wondering where these settings should go then please see Configuration. You can either set these options on your app directly or you can keep a separate module for configuration.
If you want to use a single item tuple for args, don’t forget that the constructor is a comma and not a pair of parentheses.
Using a timedelta
for the schedule means the task will
be sent in 30 second intervals (the first task will be sent 30 seconds
after celery beat starts, and then every 30 seconds
after the last run).
A crontab like schedule also exists, see the section on Crontab schedules.
Like with cron
, the tasks may overlap if the first task does not complete
before the next. If that is a concern you should use a locking
strategy to ensure only one instance can run at a time (see for example
Ensuring a task is only executed one at a time).
Available Fields¶
task
The name of the task to execute.
schedule
args
kwargs
Keyword arguments (
dict
).options
Execution options (
dict
).This can be any argument supported by
apply_async()
, e.g. exchange, routing_key, expires, and so on.relative
By default
timedelta
schedules are scheduled “by the clock”. This means the frequency is rounded to the nearest second, minute, hour or day depending on the period of the timedelta.If relative is true the frequency is not rounded and will be relative to the time when celery beat was started.
Crontab schedules¶
If you want more control over when the task is executed, for
example, a particular time of day or day of the week, you can use
the crontab
schedule type:
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
# Executes every Monday morning at 7:30 A.M
'add-every-monday-morning': {
'task': 'tasks.add',
'schedule': crontab(hour=7, minute=30, day_of_week=1),
'args': (16, 16),
},
}
The syntax of these crontab expressions are very flexible. Some examples:
Example | Meaning |
crontab() |
Execute every minute. |
crontab(minute=0, hour=0) |
Execute daily at midnight. |
crontab(minute=0, hour='*/3') |
Execute every three hours: midnight, 3am, 6am, 9am, noon, 3pm, 6pm, 9pm. |
|
Same as previous. |
crontab(minute='*/15') |
Execute every 15 minutes. |
crontab(day_of_week='sunday') |
Execute every minute (!) at Sundays. |
|
Same as previous. |
|
Execute every ten minutes, but only between 3-4 am, 5-6 pm and 10-11 pm on Thursdays or Fridays. |
crontab(minute=0, hour='*/2,*/3') |
Execute every even hour, and every hour divisible by three. This means: at every hour except: 1am, 5am, 7am, 11am, 1pm, 5pm, 7pm, 11pm |
crontab(minute=0, hour='*/5') |
Execute hour divisible by 5. This means that it is triggered at 3pm, not 5pm (since 3pm equals the 24-hour clock value of “15”, which is divisible by 5). |
crontab(minute=0, hour='*/3,8-17') |
Execute every hour divisible by 3, and every hour during office hours (8am-5pm). |
crontab(0, 0, day_of_month='2') |
Execute on the second day of every month. |
|
Execute on every even numbered day. |
|
Execute on the first and third weeks of the month. |
|
Execute on 11th of May every year. |
|
Execute on the first month of every quarter. |
See celery.schedules.crontab
for more documentation.
Starting the Scheduler¶
To start the celery beat service:
$ celery -A proj beat
You can also start embed beat inside the worker by enabling workers -B option, this is convenient if you will never run more than one worker node, but it’s not commonly used and for that reason is not recommended for production use:
$ celery -A proj worker -B
Beat needs to store the last run times of the tasks in a local database file (named celerybeat-schedule by default), so it needs access to write in the current directory, or alternatively you can specify a custom location for this file:
$ celery -A proj beat -s /home/celery/var/run/celerybeat-schedule
Note
To daemonize beat see Running the worker as a daemon.
Using custom scheduler classes¶
Custom scheduler classes can be specified on the command-line (the -S
argument). The default scheduler is celery.beat.PersistentScheduler
,
which is simply keeping track of the last run times in a local database file
(a shelve
).
django-celery also ships with a scheduler that stores the schedule in the Django database:
$ celery -A proj beat -S djcelery.schedulers.DatabaseScheduler
Using django-celery‘s scheduler you can add, modify and remove periodic tasks from the Django Admin.