Module 02 · Lesson 07
Fault tolerance and graceful degradation
Assume the parts break. Design what the user still gets.
What you will be able to do
- Distinguish fault tolerance from failover
- Apply timeout, retry with backoff, circuit breaker and bulkhead correctly
- Design an explicit degraded mode for a core feature
- Recognise when a retry worsens the failure
The four patterns that contain failure
Timeout
Every call gets a deadline derived from the caller's budget, not from a default. A dependency that is slow but alive will otherwise hold your threads until they are all gone. Fast failure frees capacity.
Retry with backoff and jitter
Retry only idempotent operations, only on errors that might succeed, and only a couple of times. Exponential backoff with randomised jitter spreads the herd so ten thousand clients do not return in the same millisecond.
Circuit breaker
After a failure threshold, stop calling for a cooldown and serve a fallback immediately. This protects both sides: your threads stay free, and the struggling dependency gets a chance to recover instead of being hammered.
Bulkhead
Isolate pools per dependency or per tenant so one saturated dependency cannot consume every connection and take the other features with it. Named after ship compartments for a reason.
The retry amplification maths
One client retrying three times turns a 1,000 requests/second dependency into 3,000 — at the exact moment it is failing. Two layers of retries give nine times the load. Retries belong at one layer, with a budget and jitter, never at every hop.
Degradation, decided in advance
Fault tolerance needs an answer for "what does the user get instead?" Write the table before the incident, because during it nobody designs calmly.
| Dependency fails | Bad behaviour | Tolerant behaviour |
|---|---|---|
| Recommendation service | Page fails entirely | Show a popular or cached list |
| Search index | Search returns 500 | Return recent and popular items with a notice |
| Personalisation | Whole feed blocked | Serve the non-personalised feed |
| Payments provider | Double charges from blind retry | Queue the payment with an idempotency key, tell the user clearly |
Two rules make this real rather than decorative: the fallback must be cheap enough to serve at full traffic, and the user must not be lied to. "Prices may be a few minutes old" is honest; showing a stale price as current is a support ticket waiting to happen.
Q. How do I choose a timeout?
Work backwards from the request budget. If a page must respond in 300 ms and this dependency is one of several, give it a slice — say 80 ms — and include the timeout in the budget. Then make sure the caller's timeout is longer than the callee's, or the caller gives up while the work continues.
Q. When is a retry actively harmful?
When the operation is not idempotent (a POST that charges a card), when every client retries simultaneously (no jitter), when the failure is deterministic (a 400 or a bad schema), or when the retry budget exceeds the caller's deadline and the work is duplicated for a response nobody will read.
Worth remembering
- Fault tolerance is about continuing to serve correctly while parts fail; failover is one mechanism that contributes.
- Retries without backoff and jitter amplify load exactly when the system can least take it.
- A timeout longer than the caller's patience is a hidden queue of dying requests.
- Degrade the feature, not the whole page. Answers beat errors; a cached answer often beats a slow truth.
Checkpoint
Answer all 2 questions to finish the lesson.
1. What does a circuit breaker do?
The breaker trips after repeated failures, fails fast for a cooldown, and then allows a trial request. This protects your capacity and gives the dependency room to recover.
2. Why does exponential backoff alone fail to stop a retry storm?
Backoff schedules the retries, but identical clients compute identical delays. Randomised jitter spreads them out; that is what turns many retries into recoverable background noise.