Module 02 · Lesson 06
Failover
A spare that nobody promotes is just a second copy of the problem.
What you will be able to do
- Compare active-passive and active-active failover
- Explain health checks, timeouts and quorum in failure detection
- Describe split-brain and how to prevent it
- State the recovery time objective for a failover design
Two shapes
| Active-passive | Active-active | |
|---|---|---|
| Capacity use | Standby idle (or read-only) | All nodes serve traffic |
| Recovery time | Detection plus promotion, seconds to minutes | Near zero if clients retry another node |
| Complexity | Moderate: one promotion path | High: conflict resolution for writes |
| Risk | Standby is stale or misconfigured | Split-brain and write conflicts |
Active-passive is the default for stateful pieces like a primary database, because it keeps one writer and therefore one truth. Active-active is worth it when the cost of any gap exceeds the cost of conflict handling — usually read paths, stateless services, and multi-region front doors.
Detect, promote, redirect
Detect
Health checks with timeouts and a failure threshold. Two consecutive misses at 2 seconds is not the same as one slow response; tune deliberately, because every retry budget you spend on detection is downtime when the failure is real.
Promote
The standby must be caught up enough to serve. State the data loss window: asynchronous replication means the last few seconds of writes may vanish, and saying that out loud is better than being asked.
Redirect
Connections must move: DNS with a low TTL, a load balancer health check, or client-side retry with a service discovery update. If clients hardcode the address, promotion changes nothing.
Prevent the double primary
A lease or quorum makes promotion require a majority. Two nodes with a stale lease each thinking they are primary will both accept writes, and reconciliation becomes a data problem instead of an availability problem.
Failover that makes things worse
Flapping is the classic: a node misses one health check under load, traffic shifts, the load on the survivor triggers more failures, and traffic bounces. Use consecutive-failure thresholds, backoff before promoting back, and hysteresis so recovery requires sustained health.
State your RTO and RPO rather than claiming "high availability": RTO is how long until service returns (the length of the failover), RPO is how much data you accept losing (the replication lag). "RTO of 30 seconds, RPO of about 2 seconds of writes" is a design statement; "it fails over automatically" is a hope.
Q. Why not fail over as fast as possible?
Because detection speed and false positives trade directly against each other. A one-second threshold turns a slow query into a failover, and repeated failovers are worse than a brief slowdown: they move load, drain replicas, and train the team to ignore alerts.
Worth remembering
- Failover has three parts: detect, promote, redirect. Skipping any one leaves a spare that does nothing.
- Detection is a trade-off: fast detection causes false positives, slow detection causes long outages.
- Split-brain — two nodes both believing they are primary — is worse than downtime because it corrupts data.
- Test failover on a schedule. An untested failover path is a hypothesis, not a capability.
Checkpoint
Answer all 2 questions to finish the lesson.
1. What are the three necessary parts of failover?
The system must notice the failure, make another node primary, and send traffic to it. A spare without detection and redirection serves nobody.
2. Why is split-brain more dangerous than an outage?
Downtime is recoverable. Two primaries writing independently create divergent state, and fixing that means choosing which writes to lose.