Dead Letter Queue (DLQ) & Retry Policy: Rescuing Stray Messages in RabbitMQ Without Stalling Workers

Has a worker ever rejected the same message over and over until CPU spiked to 100%? That is the poison message loop, a classic RabbitMQ trap. Here is how to set up a retry policy and a dead letter queue properly.

Read in Indonesian

You run a fairly busy e-commerce system. When a customer pays, the system publishes an order.paid event to a RabbitMQ queue. From there, a worker prints the shipping label, deducts warehouse stock, and emails the invoice.

Then the third-party logistics API goes down. The shipping worker fails to process the message. From that point, you start to see where the trap appears.

What does a junior developer usually do? They catch the exception (try...except), then call this magic line:

PYTHON
channel.basic_nack(delivery_tag=method.delivery_tag, requeue=True)  # ANTI-PATTERN: reject and put the message right back

The result? The message goes straight back to the front of the queue, gets picked up again by the worker in microseconds, fails again, goes back, and on it goes millions of times per minute. 😅

This phenomenon is called a poison message loop. It does not just jam the queue completely (head-of-line blocking). The CPU on the worker server also spikes straight to 100%!

So how do you handle a failed message elegantly, without blocking the rest of the queue? There are two weapons: a retry policy and a dead letter queue (DLQ).

Key takeaways

  • Three things push a message into a Dead Letter Exchange (DLX) in RabbitMQ: an explicit negative acknowledgment (basic.reject or basic.nack with requeue=false), an expired message TTL, or a queue that exceeds its capacity (x-max-length).
  • A fatal trap called the infinite poison message loop appears when a worker keeps rejecting a message with requeue=true until CPU hits 100%.
  • The exponential backoff retry architecture is built from the x-delivery-count header or custom retry metadata combined with a delayed message exchange.
  • An isolated DLQ becomes a safety net for manual inspection, an audit log, and re-playing messages after the bug is fixed.

What Is a Dead Letter Exchange, Really?

In RabbitMQ, dead lettering is not a complicated magic feature. A Dead Letter Exchange (DLX) is really just an ordinary AMQP exchange that you register on a queue through the x-dead-letter-exchange configuration argument.

A message counts as "dead" (dead-lettered) and is forwarded automatically by the RabbitMQ broker to the DLX only when one of three conditions happens.

First, the worker rejects the message explicitly with basic.reject or basic.nack and the parameter requeue=False.

Second, the message expires (TTL expired). A message has a lifetime (Time-To-Live), either at the queue level (x-message-ttl) or per individual message, and the time runs out before it is processed.

Third, the queue exceeds its limit (max length reached). The main queue has hit its maximum message count (x-max-length), so the oldest messages are dropped to the DLX.

Why Do You Need Staged Retries?

When a message fails to process, the cause usually falls into one of two categories.

There is transient failure, a temporary error: a network glitch, a database under lock contention, or a third-party API responding slowly. The fix is to try again a few seconds later (retry with backoff).

There is also permanent failure, an error that never goes away: a broken JSON payload, an invalid user ID, or a logic bug in the code. No matter how many times you retry, this message will never succeed!

The ideal strategy gives automatic retry several chances, with longer and longer gaps (say 5 seconds, 15 seconds, 60 seconds). If it still fails after 3 or 5 attempts, then the message goes to the final Dead Letter Queue (DLQ) for a developer to analyze.

The Reliable Pattern: Delayed Retry via TTL and DLX

One of the most dependable patterns in RabbitMQ without external plugins uses a middle queue (retry queue) with a TTL attached.

First, the worker consumes messages from the main queue (orders.process). On a temporary failure, the worker publishes the message to retry.exchange and records an attempt counter in the header x-retries: 1.

Second, the message lands in a retry queue (orders.retry.30s). This queue has no worker at all! It only carries x-message-ttl: 30000 (30 seconds) and x-dead-letter-exchange: orders.exchange.

Third, after 30 seconds asleep in the retry queue, RabbitMQ treats the message as expired and forwards it back to orders.process for the worker to try again.

Fourth, once the x-retries header reaches the maximum (say 3), the worker sends the message straight to orders.dlq (terminal queue). Done, no repeated drama. 😎

An Implementation in Python (Pika / Aio-Pika)

Here is an example of resilient queue declarations with Python and the pika library:

PYTHON
import json
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(host="localhost"))
channel = connection.channel()

channel.exchange_declare(exchange="dlx.orders", exchange_type="direct")  # 1. the DLX and the DLQ
channel.queue_declare(queue="orders.dlq", durable=True)
channel.queue_bind(exchange="dlx.orders", queue="orders.dlq", routing_key="order.failed")

channel.exchange_declare(exchange="orders.exchange", exchange_type="direct")  # 2. main exchange, DLX wired below

queue_args = {
    "x-dead-letter-exchange": "dlx.orders",
    "x-dead-letter-routing-key": "order.failed",
}

channel.queue_declare(queue="orders.main", durable=True, arguments=queue_args)
channel.queue_bind(exchange="orders.exchange", queue="orders.main", routing_key="order.process")

def on_message(ch, method, properties, body):  # 3. a safe worker callback
    try:
        data = json.loads(body)
        # Run the business logic (call the logistics API, for example)...
        process_shipment(data)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except TransientNetworkError:
        # Transient error: can be published to the retry queue
        print("Network timeout, routing to retry...")
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)  # Goes to the DLQ when the limit is reached
    except Exception as e:
        # Permanent error: send it straight to the DLQ, NEVER REQUEUE!
        print(f"Permanent error: {e}. Moving to DLQ.")
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)

channel.basic_consume(queue="orders.main", on_message_callback=on_message)
print("Worker siap melayani pesanan...")
channel.start_consuming()

So What Happens to Messages Stuck in the DLQ?

The DLQ is not a trash can where data gets forgotten. The DLQ is a rescue safe, and there are three things you must do.

Set up alerting. Configure a monitoring metric (say through the Prometheus RabbitMQ Exporter) to sound an alarm as soon as the number of messages in orders.dlq is greater than 0.

Inspect and run root cause analysis. A developer can look at the original payload and the error headers in the RabbitMQ Management UI to diagnose whether there is an odd payload or a new bug.

Last, run replay messages. Once the bug in the code is fixed or the third-party service is back online, you can use the rabbitmqadmin CLI or a small script to move (shovel) messages from the DLQ back to the main queue. No customer transaction gets lost!

Don't Let One Failed Message Take Down Your System

A resilient distributed system is not one that never fails. It is one that knows how to treat failure gracefully.

By replacing a careless requeue=True with a combination of retry backoff and a dead letter queue, you protect the backend worker from collapse. Every important message stays safely stored, ready to be processed again whenever the bug is fixed.

Happy messaging, and may no customer order ever drift into the void again! 👋

Inva

Writer

Get better at shipping code.

Join the other developers getting tech updates, tips, and relaxed tutorials in their inbox every week.

Stay in the loop

Tech updates, coding tips, and relaxed insights straight to your inbox every week.

No spam. Unsubscribe anytime.

Read next

See all
Struktur DataWhat a Linked List Actually Is

A linked list is a linear chain of nodes, each holding data and a pointer to the next node. Here is what that chain looks like, what it costs next to an array, and the same minimal implementation in Java, Python, and JavaScript.

For developers who learn by struggling

© 2026 Inva.dev.