Delivery guarantees, honestly stated
At-most-once, at-least-once, and exactly-once — only two of these exist on a network you don't control.
What you will be able to do
- Contrast at-most-once, at-least-once and exactly-once processing
- Spot the crash window that forces duplicates
- Design an idempotency key for a consumer
- Explain the transactional outbox pattern
Three promises
| Guarantee | What happens | Cost | Use when |
|---|---|---|---|
| At-most-once | Send and forget; may be lost | Cheapest, lowest latency | Metrics, telemetry, non-critical pings |
| At-least-once | Retry until acknowledged; may duplicate | Duplicates, needs idempotence | Almost everything user-visible |
| Effectively-once | At-least-once plus deduplication or transactions | Extra state and coordination | Money, inventory, quotas |
The middle row is where serious systems live. The third row is not a magic network property — it is a combination of retries and a consumer that refuses to apply the same event twice.
The crash window
A consumer receives a message, writes to the database, and then crashes before acknowledging. On restart the broker redelivers, because from its point of view nothing was confirmed. The write happened twice. No broker can prevent this, because the acknowledgement and the database commit are two separate systems that cannot be committed atomically.
receive(msg) begin tx insert into ledger(id, amount) values (msg.id, msg.amount) insert into processed_ids(id) values (msg.id) -- same tx deduct balance(msg.id) commit then ack(msg) -- if we crash before this, redelivery is harmless
Two lines do the work: the deduplication record is written in the same transaction as the business change, so either both exist or neither does. The retry then hits a unique constraint and is skipped. Notice the ack comes last — at-least-once tolerates extra deliveries, not lost work.
Idempotency keys
An idempotency key is a stable identifier for the intended effect, generated once at the source and carried through every retry. For an API it might be a client-generated UUID in a header. For an event stream it is the event id. The consumer stores it with a uniqueness constraint and a retention window at least as long as the redelivery window.
- Do not derive the key from mutable data. A hash of the payload changes when an unrelated field changes, and the retry becomes a new effect.
- Store the outcome, not just the key. If the first attempt returned a result, a retry should return that same result rather than re-executing.
- Expire keys deliberately. A one-hour window for a message broker that redelivers within minutes is fine; a window shorter than the retry schedule is a duplicate generator.
The transactional outbox
The dual-write problem is the sibling of the crash window. A service writes to its database and then publishes an event. If the publish fails, the database and the stream disagree forever. If the process dies between the two, downstream never hears about a change that definitely happened.
Change data capture is the same idea taken one level down: read the database's replication log and emit changes from it. It removes the relay entirely, at the cost of coupling your event schema to table structure.
Worth remembering
- The network can lose an acknowledgement after the work succeeded; the sender cannot tell the difference from a lost message.
- Most real systems are at-least-once plus idempotent consumers, and that is the correct answer.
- Exactly-once is scoped to a broker's transactional boundary, not to your side effects in another system.
- The outbox pattern is how you make a database write and an event publish succeed or fail together.