Timeouts, retries, circuit breakers and bulkheads
Most outages are a slow dependency plus unlimited patience.
What you will be able to do
- Set a timeout budget across a request path
- Apply retries with backoff and jitter without causing a storm
- Explain what a circuit breaker protects and what it hides
- Isolate dependencies with bulkheads and pool limits
The retry amplification table
Retries are the first thing engineers add and the most common cause of a self-inflicted outage. Each layer that retries multiplies the load on the layer below it.
| Layers retrying | Attempts per layer | Requests reaching the dependency |
|---|---|---|
| 1 | 3 | 3 |
| 2 | 3 | 9 |
| 3 | 3 | 27 |
| 4 | 3 | 81 |
At 81x, a dependency that was merely slow becomes unavailable, and every retry storm looks identical from the inside. The fix is to retry at one layer only — usually the outermost client — and to make every layer below it fail fast.
Timeouts are a budget
If the user waits 800 ms before abandoning the page, then the whole path — gateway, service, cache, database, third-party API — must fit inside 800 ms. Allocate it: 50 ms gateway, 20 ms auth, 600 ms service work, 100 ms slack. A dependency with a 5 s timeout inside an 800 ms budget is not protecting anything; it is allowing a thread to sit idle while the user has already left.
for attempt in 1..3: remaining = deadline - now() if remaining < 50ms: break # no budget left try call(dep, timeout=min(remaining, 300ms)) catch transient: sleep(rand(0, base * 2^attempt)) catch permanent: raise immediately # never retry 400s
Circuit breakers
A breaker watches recent failures for a dependency. Past a threshold it opens and returns failures immediately without calling the dependency; after a cool-down it lets a few probes through, and closes if they succeed. The value is not the saved call — it is the thread that is not blocked for two seconds while the dependency times out.
- Open state needs a fallback. Cached data, a default, a degraded response, or an honest error. An open breaker with no fallback is just a faster 500.
- Per dependency, not global. One shared breaker means a failing search service disables payments.
- Tune on error rate, not a single error. One timeout in a thousand requests should not trip it.
- Half-open with a small probe count. Flooding a recovering service re-opens the breaker immediately.
Bulkheads
A bulkhead gives each dependency its own bounded pool and queue. Without one, a slow third-party call consumes every worker thread, and requests that never touch that dependency also fail. This is the single most common shape of a cascading failure, and the fix is a number: at most 20 concurrent calls to the recommendation service, queue of 50, reject beyond that.
Worth remembering
- A timeout should be derived from the caller's remaining budget, not picked as a round number.
- Retries multiply traffic: three attempts across four layers is 81 requests for one user action.
- Circuit breakers convert a slow failure into a fast failure, which is what keeps threads alive.
- Every dependency needs its own bounded pool, or one slow call starves all of them.