Module 07 · Lesson 01

Consensus, quorums and the cost of agreement

Getting five machines to agree is not five times harder than one. It is a different kind of problem.

Advanced 8 min read Updated 2026-11-02

What you will be able to do

  • Compute the quorum size for a cluster and explain why odd sizes are standard
  • Explain what a Raft leader does and when a cluster stops accepting writes
  • Reason about the latency cost of cross-region quorums
  • Decide when a design actually needs consensus

Why a majority

A quorum is the smallest set of nodes that must agree for a decision to count. With majority quorums, any two quorums in the same cluster share at least one member, so a node in the overlap can carry the newest decision from one quorum to the next. That overlap is the whole trick.

N = 5   quorum = 3
tolerates = 2 failures

N = 4   quorum = 3
tolerates = 1 failure   (same as N = 3, more machines)

write quorum W + read quorum R > N  =>  read sees latest write
N = 3, W = 2, R = 2  (2 + 2 > 3)

The N=4 line explains the industry habit of odd clusters. Four nodes cost 33% more than three and buy no additional failure tolerance, because a majority of four is three.

What Raft actually does

Elect a leader
Candidates request votes with a term number. A candidate needs a majority. Split votes resolve by randomized timeouts.
Replicate the log
The leader appends entries and sends them to followers. An entry is committed once a majority has stored it.
Apply in order
Committed entries are applied to the state machine in log order, so every replica reaches the same state.
Fail over
If the leader stops heartbeating, a new election starts. Writes pause for the election timeout, typically a few hundred milliseconds to a few seconds.
Consensus is not free availability
Lose a majority and the cluster stops accepting writes by design. It refuses rather than risking two conflicting truths. If your design cannot tolerate that, consensus is the wrong tool for that data path.

Latency is geography

Same rack round trip
0.1-0.5 ms
Consensus is cheap here
Same region, 3 AZs
1-3 ms
Typical production quorum
Cross-continent
70-150 ms
Multiply by 2 for a commit round trip
Election timeout
150 ms-2 s
The write pause during failover

A three-region cluster that requires a majority for every write pays a cross-ocean round trip on the critical path — often 150 ms or more, twice if the leader must confirm with a follower before replying. The usual fix is to keep the quorum inside one region and replicate asynchronously outward, accepting that a regional disaster loses recent writes.

Do you need it?

NeedCheaper answerConsensus needed
Unique idsSnowflake ids, UUIDsNo
One writer per entityPartition ownership, sticky routingSometimes
Leader election for a jobLease in a coordination serviceYes, but delegated
Config that must not divergeVersioned config serviceYes
Money ledger orderingSingle-writer partition per accountOften no

In a design round, name the coordination service rather than inventing consensus: etcd, ZooKeeper, or the Raft implementation inside the database you already chose. Describing Raft from memory is less impressive than knowing that you only need it for a handful of keys.

Keep it to two sentences
"Leader election and cluster membership go through etcd's Raft quorum — five nodes across three AZs, so it survives one AZ. Application data does not go through consensus; it is partitioned with a single writer per key."

Worth remembering

  • Majority quorums let any two quorums overlap by at least one node, which is what makes agreement possible.
  • A five-node cluster survives two failures; a four-node cluster also survives only one.
  • Cross-region consensus pays a round trip to the majority region on every write.
  • Most features do not need consensus — they need a single writer per key, which is cheaper.

Checkpoint

Answer all 3 questions to finish the lesson.
1. How many node failures does a five-node majority quorum tolerate?
Three of five nodes still form a majority, so two can be lost. Losing three leaves only two, which cannot commit.
2. Why is a four-node cluster usually a bad choice?
Majority of four is three, so only one failure is survivable. The fourth machine adds cost and no fault tolerance.
3. A cluster spans three continents and requires a majority write quorum. What is the main consequence?
A majority must acknowledge before commit, and the majority spans distant regions, so write latency is dominated by geography.