Clocks lie, so order events by sequence
Two machines with the same wall clock will disagree by more than you think.
What you will be able to do
- Explain clock skew and why NTP does not solve it
- Choose a conflict resolution rule for a replicated field
- Use logical versions instead of timestamps for ordering
- Design an event ordering scheme that survives a failover
The skew problem
NTP keeps clocks within a few milliseconds of each other on a healthy network, which sounds acceptable until you notice that requests arrive in microseconds. Worse, a clock can step backwards when NTP corrects a drift, so a later write can carry an earlier timestamp. Two replicas can then apply writes in opposite orders and remain permanently different.
What to use instead
| Mechanism | Guarantees | Cost |
|---|---|---|
| Per-key version number | Strict order per key, compare-and-set | Requires single writer or consensus per key |
| Logical clock (Lamport) | If A caused B, A sorts before B | Only a partial order; concurrent writes still tie |
| Hybrid logical clock | Causality plus near-wall-clock reading | More state, still needs tie-break |
| Sequence from a log offset | Total order per partition | Order only inside the partition |
| Vector clock | Detects concurrency | Grows with the number of writers |
For most designs the answer is embarrassingly simple: partition by entity and keep a version counter incremented by the single owner of that partition. Order within the entity is then a number, not a guess about time.
Conflict resolution
Last-write-wins is fine for settings where the newest intent should win — a profile's theme colour. It is wrong for a balance or an inventory count, where discarding an update loses money. The rule is about the data, not the database.
Ordering across a failover
A single-writer stream is ordered until the writer dies. On failover, the new writer must continue from the committed offset, not from its own counter, or the sequence repeats. This is why log offsets beat application-assigned ids inside a stream, and why idempotent consumers matter even when ordering is guaranteed.
Worth remembering
- Wall clocks jump backwards; they are not monotonic and not comparable across machines.
- Last-write-wins is a data-loss policy unless the write is already idempotent and full-state.
- Per-entity sequence numbers give ordering without any clock agreement.
- When two writes are truly concurrent, you must pick a rule — timestamp, version, or merge.