Transactional outbox pattern: a production checklist
A database commit is not proof of event delivery. Design the relay, consumer identity and recovery tests together before moving a production workflow.
Quick answer
The transactional outbox pattern saves a business change and an event record in the same database transaction. A separate relay publishes committed events. Use it when event delivery must survive process failures, and design consumers to tolerate duplicate delivery. It provides a durable delivery obligation, not a universal exactly-once guarantee.
On this page
A transactional outbox saves a business change and an event record in the same database transaction. A relay later publishes the committed event. This closes the producer’s dual-write gap, but a relay can publish twice after an uncertain acknowledgement. Consumers still need a deliberate duplicate-handling and recovery contract. [1][7]
The order committed. Why did nothing happen next?#
Consider an illustrative order service. It commits an order and then sends OrderAccepted to a broker. The process dies between those operations. The API may already have returned success, yet the warehouse never sees an event. Reversing the order is no cure: publishing first can tell the warehouse about a transaction that later rolls back. The difficult part is not choosing which statement looks safer; it is making the obligation durable.
-- Illustrative PostgreSQL transaction. Tables are application-owned.
BEGIN;
INSERT INTO orders (id, status) VALUES ('order-example', 'accepted');
INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES (
'event-example',
'order-example',
'OrderAccepted',
'{"order_id":"order-example"}'::jsonb
);
COMMIT;The example assumes tables with suitable constraints, not a complete production schema. Both inserts use one database transaction. A failure before commit leaves neither committed row; a successful commit leaves both. A stopped relay can now discover the waiting obligation later. The outbox record is recovery state, not merely another log message. [1][6]
Conceptual illustration
One local commit, then a separate delivery
Producer transaction
Order row and outbox row
Committed obligation
Discoverable after a process crash
Relay publication
May need to repeat
Consumer effect
Must handle repeated event identity
Follow the same event through another failure. The broker accepts it, then the relay dies before recording success. On restart, the relay still finds an unfinished obligation. Retrying is the safe producer recovery, but the warehouse may see the event again. That is why the event identity must survive retries and why “the outbox guarantees exactly once” is too broad. [7]
What problem does the transactional outbox solve?#
A database update and a broker publication are separate operations. If you commit first, a process crash can prevent publication. If you publish first, the database transaction can still fail. Retrying the whole request does not explain which operation already succeeded. AWS describes this as the dual-write problem. [1]
The outbox changes the local boundary. The database stores the state change and the obligation to publish together. In PostgreSQL, a transaction groups its statements into an atomic operation; the outbox insert must actually participate in that same transaction and database, rather than run through another connection after commit. [1][6]
Our design question is therefore precise: which durable record proves that a committed change still needs delivery? If the answer is an application log line or a task held in memory, ask how recovery will find every unfinished obligation. For the broader modernization boundary, see modernizing a system that cannot stop.
Where does the guarantee end?#
The transaction guarantees the local pairing, not instant delivery. A stopped relay leaves committed events waiting. Delivery still depends on retaining the record, restoring the relay and keeping its destination usable. Define the tolerated delay before choosing a polling interval or a capture pipeline. [1]
A relay can also publish successfully and crash before recording success. Republishing is then the safe recovery choice, but it creates a duplicate. Standard Amazon SQS queues independently permit redelivery too. Treat these as normal design conditions, rather than faults that a reliable broker will necessarily remove. [7][5]
| Boundary | Possible outcome | Design requirement |
|---|---|---|
| Business transaction rolls back | No committed change or outbox obligation | Relay reads committed records only |
| Commit succeeds; relay stops | Delivery is delayed | Retain pending records and alert on their age |
| Broker accepts; relay loses acknowledgement | Publication may repeat | Preserve event identity across retries |
| Consumer commits; acknowledgement is lost | The consumer may receive the event again | Deduplicate with the local effect |
| External action succeeds; response is lost | The remote outcome is uncertain | Use provider idempotency and reconciliation |
This matrix is our review method, not a broker specification. Ask the owner of each boundary to show its recovery record. A dashboard that reports successful sends alone cannot establish whether the business effect was applied or whether a failed acknowledgement triggered another attempt.
How should you choose polling or change data capture?#
A polling relay scans pending outbox records and publishes them. A change data capture relay reads database changes instead. Debezium provides an outbox event-router transformation and uses an aggregate key for Kafka partition routing; its event identifier is distinct from that routing key. Check the connector documentation for the version you deploy. [1][2]
| Criterion | Polling relay | Change data capture relay |
|---|---|---|
| Existing operating model | Prefer when the team already owns database-backed workers | Prefer when the team already operates a capture pipeline |
| Recovery evidence | Pending records, claim ownership and retry state | Capture offsets, connector state and broker acknowledgement |
| Main review burden | Worker concurrency, scan cost and retry ordering | Source retention, connector restarts and replay |
| Choice test | Rehearse a worker crash during publication | Rehearse connector outage and recovery |
Our default is to use the delivery mechanism the team can operate and rehearse. Do not add a capture platform solely to remove a modest polling worker. Conversely, do not build custom polling merely to avoid a connector your platform team already supports. Both choices need a named owner for stalled delivery.
For polling, our recommendation is to make worker claims recoverable after a crash. An expiring lease with a unique ownership token is one option; check ownership before updating delivery state. Record success only after the destination accepts the event under its documented acknowledgement contract. A lease reduces concurrent work but cannot eliminate repeat publication after a pause or uncertain acknowledgement.
For PostgreSQL 18, logical decoding can resend changes after crash recovery, and inactive replication slots can retain resources. Include database retention and recovery in the design review. Capturing the database log does not remove the consumer replay requirement. [3]
How do consumers make retries safe?#
Use a stable event identity to detect reprocessing. For a consumer whose effect is a local database update, record the processed identity and apply the update in the same local transaction. The idempotent-consumer pattern describes storing processed message identities, including a consumer-specific identity boundary. [4]
Our implementation checklist is to enforce uniqueness on the consumer-and-event pair, handle duplicate-key outcomes deliberately, and acknowledge only after the local transaction succeeds. A separate check followed by an unprotected update leaves concurrent deliveries to race. Review the constraint and transaction together rather than trusting an application-level lookup.
The following pseudocode shows the local boundary; it is not executable SQL or a tested application implementation. Claim the identity atomically, so the operation reports whether it inserted a new record. Idempotent means that receiving the same event again does not repeat the intended local effect. [4]
BEGIN local database transaction
atomically claim (consumer_id, event_id) with a uniqueness constraint
if this transaction inserted the claim:
apply the local business update
COMMIT
acknowledge the delivery
on database failure:
ROLLBACK both the claim and the business update
leave the delivery unacknowledged for retryOur retention rule is to keep processed identities for the full supported replay window. Deleting the marker while an old event can still return allows its effect to run again. Decide whether historical replay rebuilds a separate projection or re-executes business effects, and document the supported window before cleanup runs.
Give each event a stable event_id; combine it with a consumer_id to distinguish independent subscribers. Use a separate business entity key for routing, and define any sequence or payload schema version explicitly. These are proposed application-envelope fields, not Debezium’s default column names. A delivery retry retains the original event identity. Record whether an operator is redelivering that event or authorizing a new business operation.
For remote effects, a local processed marker cannot by itself make a remote API call atomic. Our recommendation is to use the remote provider’s supported idempotency mechanism, retain its operation reference, and reconcile ambiguous outcomes. If the provider offers no recovery lookup, document that limit before enabling automatic retries.
What does the duplicate mean to the business?#
For the order example, the transport event identity and the business order identity serve different purposes. Re-delivering event-example should not reserve stock twice. A genuinely new order amendment may need a new event even though it refers to the same order. Deduplicating only by order ID can therefore suppress a legitimate later operation. Define what one event means before choosing the uniqueness constraint.
The receiving service also needs to decide whether the payload is a complete state snapshot or a delta. A newer snapshot may replace an older view under a version contract. A distinct stock increment or debit cannot be dropped just because a later event arrived first. Route ordering, processing concurrency and business ordering are related decisions, but one partition key does not prove all three.
Make operator replay explicit. Repeating delivery of the same obligation should preserve identity; authorizing a new business action should follow the command’s own audit and permission path. If a replay tool silently assigns new event IDs, it may defeat the consumer protection the team carefully designed. The recovery interface belongs in the design review alongside the transaction and relay.
Which failure tests should pass before rollout?#
Use a rehearsal environment with representative data. The following is Dopstack’s proposed acceptance checklist, not a claim that these tests have been run against your system. Keep identifiers and recovery evidence from each attempt so the result can be reviewed.
- Roll back the producer transactionForce the business update to fail and verify that no committed event is eligible for delivery. Inspect the database, not only the HTTP response.
- Stop the relay after commitConfirm that pending obligations remain discoverable. Restore the relay and reconcile delivered event identities against the committed records.
- Interrupt publication acknowledgementAllow the destination to accept an event, then interrupt the relay before success is recorded. Replay it and inspect the downstream business effect.
- Deliver concurrently to the consumerSend the same event through competing workers. Verify the uniqueness boundary and local transaction prevent repeated local effects.
- Rehearse an ambiguous remote responseSimulate a lost response after a remote action. Verify the retry uses the same provider idempotency key or enters a reconciliation workflow.
- Exercise recovery ownershipHave the on-call owner identify a stalled event, explain its state, and recover it without editing arbitrary identifiers or deleting evidence.
Our recommended dashboard tracks the age of the oldest pending obligation, retry exhaustion, quarantined events, and delivery-to-effect reconciliation. Set alert thresholds from business tolerance, not an invented universal latency target. Record who may replay an event and what evidence must be checked before doing so.
Define whether an event carries a complete state snapshot or a change to apply. An authoritative snapshot consumer may be able to ignore older versions after storing a newer one. For deltas, such as increments or debits, discarding an older distinct event can lose an operation. Specify how the contract handles missing predecessors, replay and reconciliation. Debezium’s aggregate key supports Kafka partition routing; it does not prove that downstream workers apply effects in order. [2]
When should you avoid an outbox?#
Avoid treating an outbox as a requirement to split a working application into microservices. If the whole effect can remain inside one database transaction, first ask whether messaging is necessary. The pattern earns its cost when an external obligation must survive a process failure, not merely when a diagram contains several boxes.
Keep ownership clear during a migration. Decide which system owns the write, how event contracts change, and how consumers recover before adding another publisher. A staged banking modernization approach supplies the wider migration context; this article focuses on the delivery boundary rather than banking-specific implementation advice.
Transactional outbox questions#
- Does a transactional outbox guarantee exactly-once delivery?
- No. The producer transaction pairs business state with an event record. Relays and brokers can deliver again, so consumers must tolerate repetition. Review the local effect and any remote operation separately before making an exactly-once claim. [1][4][5]
- Is an outbox the same as event sourcing?
- No. An outbox records events awaiting publication alongside business state. It does not require those events to become the authoritative history used to rebuild that state. Choose event sourcing only after separately evaluating the application’s storage and recovery model. [1]
- Do I need Kafka and Debezium?
- No. A polling relay can publish committed outbox records without Debezium. Debezium offers an event-router transformation for capture-based implementations. Choose based on the operating model you can support, rather than assuming that the pattern requires a particular broker. [1][2]
- What should a failed-event replay preserve?
- Our recommendation is to preserve the original event identity, payload contract and audit context for a transport redelivery. Distinguish that operation from creating a new business command. This lets the consumer apply its existing duplicate-handling rules deliberately.
