Celery Task Routing and Reliability

How you split the queues decides throughput, and when you send the acknowledgement decides whether you lose work or run it twice.

When Everything Goes Into One Queue

Celery is the distributed task queue most widely used in Python: a producer puts a message on a broker and a worker pulls it off and runs it. The default configuration puts every task on a single queue. When sending an email and encoding a video stand in the same line, a video encode that takes tens of seconds blocks the short mail job behind it.

The starting point is to split queues by the nature of the work. Put short, frequent tasks and long, heavy tasks on different queues, and attach separate workers to each. Then a backlog on one side does not spread its delay to the other.

How an Exchange Routes Messages to Queues

Queue separation is implemented as routing rules. Celery follows the Advanced Message Queuing Protocol (AMQP) model, where a message does not go straight to a queue but passes through an exchange. The exchange looks at the routing key and the binding rules to decide which queue receives it.

Exchanges come in four kinds by routing behavior, and the kind you use determines the rule for picking a queue.

In a topic exchange, feed.# is a binding pattern that accepts every key beginning with feed.. When publishing you use a concrete key such as feed.sync. Queue definitions and routing rules are configured like this.

from celery import Celery
from kombu import Exchange, Queue
 
app = Celery('myapp')
 
default_exchange = Exchange('tasks', type='direct')
feed_exchange = Exchange('feeds', type='topic')
 
app.conf.task_queues = (
    Queue('default', exchange=default_exchange, routing_key='celery'),
    Queue('priority', exchange=default_exchange, routing_key='priority',
          queue_arguments={'x-max-priority': 10}),
    Queue('feed_tasks', exchange=feed_exchange, routing_key='feed.#',
          durable=True),
)
 
app.conf.task_routes = {
    'myapp.tasks.high_priority': {'queue': 'priority', 'routing_key': 'priority'},
    'feeds.*': {'queue': 'feed_tasks', 'routing_key': 'feed.sync'},
}

task_routes maps task name patterns to queues and supports both wildcards and function-based routers. You can also name the queue directly in code with apply_async(queue=...), but keeping the rules in configuration means the call site does not need to know queue names.

Ack Timing Trade-offs

Routing is a throughput problem; acknowledgement (ack) is a reliability problem. A worker sends an ack to the broker to say it has processed the message, and the broker removes the message from the queue only once that ack arrives. When the ack is sent splits into two options.

Early ack, the default, sends the ack as soon as the message is received. If the worker dies mid-execution, that task is gone. Late ack sends the ack after execution completes, so on a crash the broker redelivers the same message to another worker.

app.conf.task_acks_late = True
app.conf.task_reject_on_worker_lost = True
app.conf.worker_prefetch_multiplier = 1

task_acks_late=True prevents loss, and task_reject_on_worker_lost=True requeues the message even when the worker terminates abnormally. The price is that duplicate execution has to be accepted.

Absorbing Duplicate Execution With Idempotency

Turning on late ack makes the delivery guarantee at-least-once, which means the same message can run twice. Duplication is normal in a distributed queue, and it has more than one cause.

A Redis broker tracks unacknowledged delivery through a visibility timeout rather than RabbitMQ's AMQP delivery state. It redelivers a message that is not acknowledged within the default 3,600 seconds (Celery Redis documentation). A task that runs longer can execute twice.

app.conf.broker_transport_options = {
    'visibility_timeout': 3600,  # Redis only, in seconds
}

Design for idempotency so that repeated execution leaves one external effect. A database unique constraint is insufficient when a task calls an external system such as a payment processor. The processor must also promise to return the same charge result for the same idempotency key.

function process_payment(order_id, idempotency_key):
    payment = database.get_or_create_payment(
        unique_key=idempotency_key,
        order_id=order_id,
        status="pending"
    )
 
    charge = payment_provider.charge(
        amount=payment.amount,
        idempotency_key=idempotency_key
    )
 
    database.mark_succeeded(
        unique_key=idempotency_key,
        provider_charge_id=charge.id
    )
    return database.get_payment(unique_key=idempotency_key)

If the worker dies after the external approval but before mark_succeeded, the local row remains pending. A redelivered task sends the same key, so the processor returns the earlier result instead of creating another charge. A processor without that contract requires separate lookup, reconciliation, and compensation paths. Celery likewise recommends idempotent tasks because a late-ack task may execute more than once (Celery task documentation).

Retries rest on the same premise. retry_backoff=True raises the delay ceilings as 1, 2, and 4 seconds, while default jitter randomizes the actual delay within each ceiling (Celery task documentation). Retrying a non-idempotent task can accumulate partial effects.

@app.task(bind=True, autoretry_for=(Exception,), retry_backoff=True, max_retries=5)
def fetch_url(self, url: str) -> str:
    import requests
    r = requests.get(url, timeout=10)
    r.raise_for_status()
    return r.text[:2000]

Prefetch and Serialization

A worker fetches several messages ahead of time. worker_prefetch_multiplier sets how many, and the default of 4 is tuned for short tasks. For long tasks, lower it to 1 so that work reserved by one worker does not starve the others.

Serialization is directly a security matter. The default, JSON, carries only data and is safe. Pickle carries Python objects as-is and creates a risk of arbitrary code execution, so block it with accept_content=['json'] and do not run workers as root. Prefetch and serialization govern throughput and safety respectively, but they belong in the same decision as routing, ack, and idempotency.

Summary

Queue separation decides throughput, and ack timing decides reliability. Preventing loss with late ack means accepting duplicate execution in exchange, and idempotency plus exponential-backoff retries are what absorb that duplication. With Redis, the visibility timeout becomes another source of duplication on long tasks, so tune it together with expected task duration. Routing, ack, idempotency, prefetch, and serialization are not independent settings but one reliability bundle to be decided together.