Celery Architecture and the Message Lifecycle

In Celery, a single message goes through a fixed sequence of stages from the producer publishing it to the worker executing it.

The four core roles

Celery is a message-based distributed task queue. A producer turns work into a message and publishes it to the broker, and a worker takes that message off the queue and runs it. In this task queue pattern, the publishing side and the executing side are never directly connected.

The arrows in the diagram show the one-way path a message travels. The table below lists what each role does.

The producer publishes messages without knowing the state of any worker. Workers consume messages from the broker on their own. That is why adding workers watching the same queue scales throughput horizontally.

The flow of one message

A single task call splits into client-side publication and worker-side reception. The client goes through routing, serialization, and publication in order. The worker handles reception, request construction, execution, and acknowledgement in that order.

In this flow the client gets control back immediately after publication. If the result is needed, it queries the result backend later.

The protocol v2 message structure

Protocol version 2 is the serialization convention Celery uses to build a message. It splits the message into three parts: properties, headers, and body. Headers can be read without deserializing the body.

message = {
    'properties': {
        'correlation_id': task_id,            # same as task_id
        'content_type': 'application/json',   # serialization format
        'content_encoding': 'utf-8',
    },
    'headers': {                              # accessible without unpacking the body
        'lang': 'py',
        'task': 'myapp.tasks.add',            # normalized task name
        'id': task_id,
        'root_id': root_id,                   # for workflow tracing
        'retries': 0,                         # retry count
        'eta': None,                          # scheduled execution time
        'expires': 600,                       # expiry (seconds)
    },
    'body': (
        args_tuple,                           # positional arguments
        kwargs_dict,                          # keyword arguments
        {'callbacks': [], 'errbacks': [], 'chain': [], 'chord': None},
    ),
}

Separating headers from the body is the key change in v2. Brokers and monitoring tools handle routing and tracing from headers alone, without unpacking the body. The header's task says which function to run, and id says where to match the result.

The broker and the AMQP model

The internal model by which a broker puts messages into queues and takes them out follows AMQP (Advanced Message Queuing Protocol). The producer sends a message to an exchange, and the exchange distributes it to queues according to binding rules.

The exchange type decides the distribution scheme. Direct sends to the queue whose routing_key matches exactly, topic uses wildcard patterns, and fanout sends to every queue.

RabbitMQ is a broker that implements this AMQP model natively. Redis imitates the same semantics through Kombu, Celery's messaging abstraction library. On Redis a queue becomes a list (LPUSH/BRPOP), and QoS is replaced by a visibility timeout.

The table gives general criteria and is not absolute. If routing and reliability matter, RabbitMQ fits; if simplicity of setup comes first, Redis is fine.

Worker reception and bootsteps

A worker does not receive messages the moment it starts. A blueprint, which is a bootstep dependency graph, turns components on in order. Message reception begins in the Tasks step inside the final Consumer.

The final Consumer step of the worker blueprint expands into the consumer blueprint below it.

In the Tasks step, kombu.Consumer subscribes to queues and attaches an on_message callback. When a message arrives, the task name in the header selects a strategy, which builds a request and hands it to the pool.

def on_message(body, message):
    task_name = message.headers['task']     # 1. task name from the header
    strategy = strategies[task_name]        # 2. look up the per-task strategy
    request = strategy(                      # 3. decode + build the Request
        message,
        accept=consumer.accept,
        timeout=worker.pool_timeout,
    )
    dispatch_to_pool(request)               # 4. dispatch to the execution pool

strategy is a handler function precompiled for each task. Request is an object that unpacks the header and body into execution information such as args, kwargs, and eta. Once that object enters the pool, the actual function is called.

Choosing a worker pool

The worker pool is what actually runs tasks. The concurrency model and the kind of work each pool suits differ.

If the work is CPU-heavy or memory leaks are a concern, prefork is the safe choice. For I/O work with frequent external calls, gevent or eventlet deliver high concurrency with little memory. For development and debugging, the single-process solo pool is convenient.

Ack strategy and prefetch

Whether messages are lost is decided by when the acknowledgement (ack) is sent. The default is early ack: the worker acks the broker as soon as it receives the message. If the worker dies mid-execution, that work disappears.

Setting task_acks_late=True sends the ack after execution finishes. If the worker dies partway, the message stays in the queue and another worker runs it again. The same work can therefore run twice, so this is safe only for idempotent tasks.

Prefetch is the number of messages a worker reserves in advance at one time. A larger worker_prefetch_multiplier raises throughput but concentrates work on one worker. For long-running tasks, lowering it to 1 distributes work evenly.

app.conf.update(
    task_acks_late=True,              # ack after execution completes
    task_reject_on_worker_lost=True,  # requeue the message if the worker dies
    worker_prefetch_multiplier=1,     # reserve only one at a time
)

Used together, these three settings keep work alive when a worker dies and keep the queue from piling up on one side.

Summary

Celery's behavior comes down to four roles, producer, broker, worker, and result backend, and the flow of a single message between them. The client builds a message with protocol v2 and publishes it to an exchange, and the broker holds it in a queue according to the AMQP model. The worker boots in blueprint order, then receives messages through the consumer and runs them in the pool.

Reliability is tuned through the combination of ack strategy, prefetch, and retry settings. For the broker, choose RabbitMQ when routing and reliability matter and Redis when simplicity comes first. Deeper configuration values are available in the configuration reference in the official documentation (docs.celeryq.dev).