This document describes Celery 2.3. For development docs, go here.

celery.concurrency.processes

Process Pools.

class celery.concurrency.processes.TaskPool(limit=None, putlocks=True, logger=None, **options)

Process Pool for processing tasks in parallel.

Parameters:
  • processes – see processes.
  • logger – see logger.
limit

The number of processes that can run simultaneously.

logger

The logger used for debugging.

class Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, timeout=None, soft_timeout=None)

Class which supports an async version of the apply() builtin

class Process(group=None, target=None, name=None, args=(), kwargs={})

Process objects represent activity that is run in a separate process

The class is analagous to threading.Thread

authkey
daemon

Return whether process is a daemon

exitcode

Return exit code of process or None if it has yet to stop

ident

Return identifier (PID) of process or None if it has yet to start

is_alive()

Return whether process is alive

join(timeout=None)

Wait until child process terminates

name
pid

Return identifier (PID) of process or None if it has yet to start

run()

Method to be run in sub-process; can be overridden in sub-class

start()

Start child process

terminate()

Terminate process; sends SIGTERM signal or uses TerminateProcess()

class TaskPool.Pool.ResultHandler(outqueue, get, cache, poll, join_exited_workers, putlock)
body()
exception TaskPool.Pool.SoftTimeLimitExceeded

The soft time limit has been exceeded. This exception is raised to give the task a chance to clean up.

class TaskPool.Pool.Supervisor(pool)
body()
class TaskPool.Pool.TaskHandler(taskqueue, put, outqueue, pool)
body()
class TaskPool.Pool.TimeoutHandler(processes, cache, t_soft, t_hard)
body()
TaskPool.Pool.apply(func, args=(), kwds={})

Equivalent of apply() builtin

TaskPool.Pool.apply_async(func, args=(), kwds={}, callback=None, accept_callback=None, timeout_callback=None, waitforslot=False, error_callback=None, soft_timeout=None, timeout=None)

Asynchronous equivalent of apply() builtin.

Callback is called when the functions return value is ready. The accept callback is called when the job is accepted to be executed.

Simplified the flow is like this:

>>> if accept_callback:
...     accept_callback()
>>> retval = func(*args, **kwds)
>>> if callback:
...     callback(retval)
TaskPool.Pool.close()
TaskPool.Pool.grow(n=1)
TaskPool.Pool.imap(func, iterable, chunksize=1)

Equivalent of itertools.imap() – can be MUCH slower than Pool.map()

TaskPool.Pool.imap_unordered(func, iterable, chunksize=1)

Like imap() method but ordering of results is arbitrary

TaskPool.Pool.join()
TaskPool.Pool.map(func, iterable, chunksize=None)

Equivalent of map() builtin

TaskPool.Pool.map_async(func, iterable, chunksize=None, callback=None)

Asynchronous equivalent of map() builtin

TaskPool.Pool.shrink(n=1)
TaskPool.Pool.terminate()
TaskPool.grow(n=1)
TaskPool.on_start()

Run the task pool.

Will pre-fork all workers so they’re ready to accept tasks.

TaskPool.on_stop()

Gracefully stop the pool.

TaskPool.on_terminate()

Force terminate the pool.

TaskPool.shrink(n=1)
TaskPool.terminate_job(pid, signal=None)

Previous topic

celery.concurrency.solo

Next topic

celery.concurrency.processes.pool

This Page