Module 04 · Lesson 04

Replication: copies, lag, and the read you cannot trust

The replica is not a backup, and the seconds it lags are where stale data bites.

Advanced 9 min read Updated 2026-11-01

What you will be able to do

  • Compare leader-follower, multi-leader, and leaderless replication
  • Diagnose stale reads caused by replication lag
  • Design a read-your-own-writes guarantee
  • Explain failover, promotion, and split brain

Three topologies

TopologyHow writes flowFailure behaviour
Leader-followerAll writes to one leader, replicated to followersLeader dies: promote a follower. Followers die: reads come from the rest
Multi-leaderSeveral leaders accept writes, replicate to each otherLeads to write conflicts that must be resolved
Leaderless (quorum)Client writes to several replicas, reads from severalNo promotion needed; consistency set by R + W > N

Leader-follower is the default for application databases because it keeps writes simple. Multi-leader exists when regions must accept writes while a cross-region link is unreliable — and it forces you to answer: which write wins when two users edit the same record at once?

A replica is not a backup
A replica faithfully applies a bad delete. Backups must be independent, point-in-time, and restored in a drill you have actually run. Interviewers notice when someone says backup and means replica.

Lag is a product decision

Replication is asynchronous by default. If the leader commits and fails before shipping the change, the follower never sees it — the write is lost. Synchronous replication closes that gap but makes writes wait for a remote round trip.

Same-region async lag
1-50 ms
Usually invisible
Cross-region async lag
80-300 ms
Enough to break instant read-after-write
Semi-sync commit
+1 RTT to a replica
Durability at a latency price
Failover decision window
10-30 s
Health checks plus promotion, typically

"Read from replicas" sounds free and is not. The user posts a comment, the redirect goes to a page served by a lagging replica, and their comment is missing. That is the classic read-your-own-writes bug, and it is a design bug, not a caching quirk.

Pin briefly
After a write, route that user's reads to the leader for a few seconds or until lag catches up. Simple, and cheap when writes are a small share of traffic.
Track a version
Return a commit LSN or version with the write. Reads wait until a replica has reached it, or fall back to the leader.
Read your own data locally
Render the just-written object from the client or the write response, so the next read is not load-bearing.
Accept it, and say so
For likes and view counts, stale reads are fine. Name the tolerance explicitly instead of pretending replication is instant.

Failover and the split-brain problem

Detect
The failure detector must distinguish a dead leader from a slow network. Too eager and you promote a replica while the leader is alive and still serving writes.
Fence
Revoke the old leader's ability to write — using a fencing token or a quorum requirement — before promoting.
Promote
Pick the follower with the most complete log, promote it, and repoint the fleet. Replicas that were behind must resync, often by snapshot.
Repair
When the old leader returns, it must rejoin as a follower and discard or reconcile divergent writes rather than resume as leader.
Split brain
Two leaders accepting writes to the same key produce two histories. Merging them later is often impossible without data loss. This is why production failover uses quorum-based consensus rather than a heartbeat plus a hope.

How to say it out loud

"Primary takes writes; three replicas serve reads in the same region. Posts are read-your-own-writes, so the writer is pinned to the primary for two seconds after publishing. Timelines tolerate roughly one second of staleness. Failover promotes a replica through the orchestrator and repoints the service; the old primary rejoins as a replica." That paragraph answers availability, consistency, and failure in one breath.

Worth remembering

  • Replication buys read scale and availability, and pays with lag and failover complexity.
  • A replica can be behind by milliseconds or minutes; the design must state what staleness is acceptable.
  • Read-your-own-writes is usually solved by pinning the user to the leader briefly or tracking a version.
  • Split brain happens when two nodes both believe they are leader; fencing tokens and quorum promotion prevent it.

Checkpoint

Answer all 3 questions to finish the lesson.
1. A user posts a comment and immediately reloads to find it missing. Most likely cause?
Asynchronous replication means a follower can serve a read from before the commit. Pinning the writer's reads to the primary briefly is the standard fix.
2. Why is a read replica not a backup?
Replication copies operations, not history. Independent point-in-time backups are what let you recover from corruption or an accidental delete.
3. What does a fencing token prevent during failover?
The token establishes which leader is current, so the superseded one is rejected by the storage layer — avoiding two writers on the same data.
← PreviousTransactions, isolation, and the anomalies in between