Event patterns that survive contact with production
Events are a decoupling tool and a debugging tax. Use three patterns, not thirteen.
What you will be able to do
- Tell an event apart from a command and know why the difference matters
- Design a saga with compensations for a multi-service workflow
- Version an event schema without breaking consumers
- Handle poison messages and replay safely
Events, commands, and the difference
| Event | Command | |
|---|---|---|
| Tense | Past: order_placed | Imperative: charge_card |
| Owner | The producer that owns the fact | The caller that wants something done |
| Consumers | Unknown, many | One intended handler |
| On failure | Consumer retries; fact stays true | Caller must learn it failed |
| Coupling | Producer knows nothing downstream | Caller knows the handler's contract |
The distinction is practical. If a topic is named like a command and consumed by five services, every consumer is implicitly part of the caller's transaction, and nobody can change the workflow without a coordinated deploy. Name facts as facts and keep command channels one-to-one.
Fan-out: one event, many readers
Placing an order should notify inventory, payments, email and analytics. With a log, that is four consumer groups and no producer change. With a queue, it is four queues and a fan-out step that someone must maintain. The log wins on cost of adding a consumer, which is exactly the thing that changes most often.
Sagas: no distributed rollback
A workflow spanning payments, inventory and shipping has no single transaction. A saga runs local transactions in order and, on failure, runs compensating actions that undo the business effect: refund the card, release the reservation. Compensations are not rollbacks — an email cannot be unsent, so order the steps so irreversible actions happen last.
Sagas are eventually consistent by construction. Between charge and confirm, a user may see "processing". That state must exist in your API and in your database — a saga without visible intermediate states produces support tickets that look like data loss.
Schemas and poisoning
- Add optional fields only. Consumers ignore unknown fields; they break on renamed or removed ones.
- Never change a field's meaning. A
pricethat switches from cents to dollars is a silent, expensive corruption. - Include a schema version and an event id. Version for routing, id for deduplication.
- Keep payloads small. Large blobs belong in object storage with a reference in the event.
A poison message is one that always fails — malformed, or referencing a deleted entity. Retrying it forever blocks the partition. Send it to a dead-letter topic after a few attempts with the failure reason attached, alert when the dead-letter rate moves, and provide a replay tool that re-injects fixed messages. Without the replay tool, dead-letter queues are write-only graveyards.
Worth remembering
- An event states a fact in the past tense; a command requests an action. Mixing them couples services through a topic name.
- Fan-out is cheap on a log and expensive on a queue you have to copy.
- A saga is a sequence of local transactions with explicit compensations — there is no distributed rollback.
- Schema changes are additive or they are breaking; adding an optional field is the only free move.