Transactional Outbox Pattern: Database and Message Broker Consistency Without 2PC

Seen an order land in the database while the confirmation email never goes out? That is the dual-write problem. Here is how the Transactional Outbox Pattern stops your data and your events from disagreeing.

Read in Indonesian

There is one lopsided pattern in backend code that almost everyone has written without noticing. Save the order to the database first, then publish the event to the message broker. Looks tidy, doesn't it?

PYTHON
async def create_order(order_data):  # Scenario 1: save to the DB first, publish the event after
    async with db.transaction():
        order = await db.insert_order(order_data)

    # What happens if this line crashes or hits a network timeout?
    await message_broker.publish("order.created", order.dict())

Worried that the message might not get sent, some of us then decide to reverse the order.

PYTHON
async def create_order(order_data):  # Scenario 2: publish the event first, save to the DB after
    await message_broker.publish("order.created", order_data)

    # What happens if the database transaction fails (constraint violation)?
    async with db.transaction():
        await db.insert_order(order_data)

Both store the same time bomb, called the Dual-Write Problem.

In a distributed system, you really cannot guarantee that two separate network operations succeed together: writing to a SQL database and publishing an event to a message broker like Apache Kafka or RabbitMQ. Not without a dedicated consistency mechanism.

If scenario 1 fails at publish, the order is already sold in the database, but the confirmation email and the warehouse instructions never fire. In scenario 2, if the database commit fails, the event has already spread across the whole microservices ecosystem even though the order never existed! 😬

The theoretical answer used to be Two-Phase Commit (2PC), also known as XA Transactions. In the modern cloud world, though, 2PC is often avoided: slow, fiddly, and some modern message brokers refuse to support it.

So what is the solution? This is where the Transactional Outbox Pattern became the industry gold standard.

Key takeaways

  • The dual-write problem appears because writing to the database and publishing to a message broker are two separate network operations that cannot be guaranteed to succeed together.
  • The Transactional Outbox Pattern parks the event in an outbox table inside the same database transaction, so its fate becomes atomic: all saved, or nothing at all.
  • The outbox table can be drained by a Polling Publisher at small to medium scale, or by Change Data Capture (CDC) at thousands of transactions per second.
  • Delivery is guaranteed at-least-once, so downstream consumers must be idempotent or twin events get executed twice.

Riding the Database's ACID Transaction

The key to the Outbox Pattern is simple: do not talk to two systems at once inside one business request flow. Talk only to your own local database!

Relational databases like PostgreSQL or MySQL already have solid ACID transactions, tested for decades. If you can slip the event you want to send into the same database transaction as the business data, you get an atomic guarantee: everything is saved, or nothing is.

It is like writing two things on one shopping receipt. If the receipt tears in the middle, neither can be read. There is no way one survives while the other vanishes.

The flow is elegant:

  1. Create an extra table named outbox in the same database as your business tables (say the orders table).
  2. When a create_order request arrives, the app opens a database transaction:
    • Insert a new row into the orders table.
    • Insert the event payload you want to send into the outbox table.
    • Run COMMIT.
  3. A separate component (Outbox Relay / worker) reads unsent rows from the outbox table, publishes them to the message broker, then marks those rows processed or deletes them.

What Goes Inside the Outbox Table?

Here is a simple DDL schema for an outbox table in PostgreSQL:

SQL
CREATE TABLE outbox_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(64) NOT NULL, -- example: 'Order'
    aggregate_id VARCHAR(64) NOT NULL,   -- example: 'ord-12345'
    event_type VARCHAR(64) NOT NULL,     -- example: 'OrderCreated'
    payload JSONB NOT NULL,              -- event payload in JSON format
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    processed_at TIMESTAMPTZ NULL        -- timestamp when the publish succeeds
);

CREATE INDEX idx_outbox_unprocessed ON outbox_events (created_at) WHERE processed_at IS NULL;

With that partial index (partial index), the lookup for unprocessed events stays fast without a full table scan.

Two Ways to Drain the Outbox: Polling vs CDC

How do you move data from the outbox table to the message broker? Two approaches are common.

Polling publisher, for small to medium scale

A simple background worker can be a goroutine in Go, a Celery/APScheduler worker in Python, or a periodic scheduler. It runs a polling query every few seconds:

PYTHON
async def process_outbox():  # Fetch a batch of unsent outbox rows with a row-level lock
    async with db.transaction():
        events = await db.fetch(
            """
            SELECT id, event_type, payload
            FROM outbox_events
            WHERE processed_at IS NULL
            ORDER BY created_at ASC
            LIMIT 50
            FOR UPDATE SKIP LOCKED
            """
        )

        for event in events:
            await kafka_producer.send(topic=event['event_type'], value=event['payload'])
            await db.execute(
                "UPDATE outbox_events SET processed_at = NOW() WHERE id = $1",
                event['id']
            )

Using FOR UPDATE SKIP LOCKED lets you run several outbox worker instances in parallel without them fighting over the same rows (race condition).

Change Data Capture, for massive scale

When transaction volume reaches thousands per second, constant polling of the database starts to weigh on database CPU.

Instead, use a CDC tool like Debezium. Debezium reads the Write-Ahead Log directly (WAL in Postgres, binlog in MySQL) at the disk level. The moment a new row lands in the outbox table, Debezium catches the change and forwards it to a Kafka topic in real-time with sub-second latency, without loading your database's query engine!

The Consumer Must Be Idempotent, No Exceptions

The Outbox Pattern only guarantees At-Least-Once delivery. That means the message is guaranteed not to be lost, but it can be delivered more than once if there is a crash between the publish and the status update.

That is why every consumer in the downstream microservices must be designed to be idempotent:

  • Use the unique event id from the outbox as an idempotency key.
  • Store the IDs of events already processed in a Redis cache or a processed_events table on the consumer side.
  • If an event with the same ID arrives again, ignore it and send an ACK right away.

So Why Move Away from Dual-Write?

Building reliable microservices architecture is not about avoiding network failure. It is about designing a system that stays consistent when network failure, which cannot be avoided, actually happens.

With the Transactional Outbox Pattern, you can sleep well without wondering where order data went or why a customer's balance got deducted twice. Both either happen together, or not at all. Tidy, right? 🎉

Happy outboxing, and may your events always arrive on time even when the broker is sulking! 👋

Inva

Writer

Get better at shipping code.

Join the other developers getting tech updates, tips, and relaxed tutorials in their inbox every week.

Stay in the loop

Tech updates, coding tips, and relaxed insights straight to your inbox every week.

No spam. Unsubscribe anytime.

Read next

See all
Struktur DataWhat a Linked List Actually Is

A linked list is a linear chain of nodes, each holding data and a pointer to the next node. Here is what that chain looks like, what it costs next to an array, and the same minimal implementation in Java, Python, and JavaScript.

For developers who learn by struggling

© 2026 Inva.dev.