Start with the dual-write failure
An order update commits successfully, but publishing OrderConfirmed fails. The warehouse never sees the order. Reverse the sequence and a different problem appears: the message is published, then the database transaction rolls back. The warehouse sees an order that does not exist.
The transactional outbox pattern places the business mutation and the event record in the same database transaction. A separate relay publishes committed records. Its guarantee is atomic publication intent, not atomic delivery to every downstream system. The pattern's original description explicitly calls out duplicate publication and the need for idempotent consumers. Transactional outbox pattern
The design below extends that boundary with explicit sequencing and replay rules. Those rules are application design recommendations; the broker cannot infer them from a JSON payload.
Define the aggregate before defining the topic
An aggregate is the smallest business object whose transitions must remain ordered: an order, subscription, or document version. Give each aggregate a monotonically increasing version allocated within the transaction that updates it. A timestamp is inadequate when two changes occur together or clocks disagree.
An illustrative event envelope is:
{
"event_id": "evt_7c45",
"aggregate_type": "order",
"aggregate_id": "ord_204",
"aggregate_version": 18,
"schema_version": 2,
"event_type": "order.confirmed",
"correlation_id": "checkout_91",
"payload": { "order_id": "ord_204" }
}
The aggregate version expresses business order. The schema version describes payload interpretation. The correlation ID connects a wider workflow. None replaces the event ID used to identify a particular publication.
Put a unique constraint on aggregate identity plus aggregate version. Update the aggregate with a lock or optimistic version predicate, and insert the event before committing. Never calculate the next version with an unlocked MAX(version) + 1 query.
Choose a relay with an explicit recovery model
| Relay | Operational advantage | Failure boundary to address |
|---|---|---|
| Database poller | Simple deployment and inspection | Publish succeeds but marking sent fails |
| Change data capture | Reads committed database changes | Connector offsets, log retention, and bootstrap |
| Application job after commit | Low initial complexity | Process dies before durable job publication |
A poller can claim work in short transactions, publish outside the transaction, and finalize with a lease token. If the lease expires, a replacement worker may publish again. Consumers must tolerate that. For strict aggregate ordering, prevent later versions from overtaking earlier unpublished versions or make the consumer buffer gaps.
Debezium's outbox router uses the aggregate ID as the message key by default, which supports order within Kafka partitions. That is a partitioning mechanism, not a global ordering guarantee or a substitute for correct producer sequencing. Debezium outbox event router
Make consumer state advancement atomic
A consumer should commit its projection update, processed-event marker, and aggregate checkpoint together. If those records live in separate databases, the same dual-write problem reappears on the receiving side.
For a sequential projection, compare incoming version v with checkpoint c. If v is the next expected version, apply it. If it is an already recorded event, acknowledge without repeating the effect. If it creates a gap, park it and request recovery. A lower version with an unknown event ID deserves investigation; blindly discarding it can hide corruption or an invalid replay.
Some projections can replace their entire state from a newer authoritative snapshot. Others, such as accounting deltas, must process every transition. Document which kind each consumer implements. “Last event wins” is a design choice with consequences, not a universal solution.
Replay needs a separate operating mode
A replay that rebuilds a search index should not resend customer emails. Classify consumers as projection builders or external-effect executors. Give a replay a run ID, selected event range, target projection version, and explicit policy for external effects.
Build a replacement projection beside the live one. Compare record counts, domain totals, missing identifiers, and representative queries before switching readers. A replay is not proven correct because the queue drained.
Keep event schemas compatible with retained history. Additive fields need defaults; changed meanings may need an upcaster or a new event type. Test the oldest retained schema against the newest consumer before deployment.
Operate the oldest unhandled event
Monitor the age of the oldest unpublished event, sequence gaps, duplicate rate, consumer lag, and poison-event age. Alert on business delay thresholds rather than one universal queue-length limit. Ten stalled orders may matter more than a million optional analytics events.
Inject a relay crash immediately after broker acknowledgment. Run two consumers against the same event. Replay an old schema into a new projection. Introduce a missing aggregate version and verify that later state cannot silently advance past it.
Use an outbox when a committed change must reliably produce asynchronous work. Avoid adding one to every database write by habit. Its value comes from an explicit delivery obligation and a recovery path that operators can understand.