Seven anti-patterns that quietly cost a fortune
None of these are bugs. All of them multiply traffic, latency or storage until the bill arrives.
What you will be able to do
- Recognise chatty and over-fetching interfaces from their symptoms
- Bound fan-out and parallelise independent calls
- Explain why synchronous chains amplify tail latency
- Identify hot keys and retention leaks before they become incidents
1-2. Chattiness and synchronous chains
A chatty interface makes many small calls where one would do. The per-call cost is not the work, it is the network and serialisation. At 0.5 ms per same-region call, 400 calls is 200 ms before any real work happens.
Sequential chains are the same problem with a tail-risk bonus. If five independent calls each have a p99 of 100 ms, running them sequentially gives a p99 near 500 ms; in parallel, near the slowest one. The fix is structural: identify independent calls, run them concurrently with a shared deadline, and keep the dependent ones in order.
sequential: 60 + 60 + 60 + 60 + 60 = 300 ms p50, tails add parallel: max(60, 60, 60, 60, 60) = ~60-80 ms p50, one tail
3. Unbounded fan-out
A user with two million followers posts. Fan-out on write creates two million timeline entries; fan-out on read queries two million followees. Neither is acceptable as stated, which is why real systems use a hybrid: fan-out on write for normal accounts, pull-on-read for very large accounts, merged at request time.
| Approach | Writes | Reads | Breaks at |
|---|---|---|---|
| Fan-out on write | Expensive, proportional to followers | Cheap, precomputed | Celebrity accounts |
| Fan-out on read | Cheap | Expensive, merge many sources | Heavy readers |
| Hybrid | Cheap for celebrities | Merge push and pull | Complexity |
4. Over-fetching
- **SELECT \* on a wide table** pulls blobs you never render. Select columns, or split large fields into their own table.
- Fetching a list to show a count does the work of a full scan. Use a count query or a maintained counter.
- Returning full objects to mobile clients costs bandwidth and parse time. Return what the screen needs.
- Serialising everything to JSON twice — once in the service, once in the gateway — doubles CPU for no benefit.
5. Hot keys
A cache or partition keyed by a popular entity concentrates all traffic on one shard. Symptoms: one CPU core saturated while others idle, one partition's lag climbing, one Redis node near its network limit. Fixes are to split the key with a random suffix and merge on read, replicate the hot value to every node, or add a local in-process cache in front of the shared one.
6-7. Retry storms and forever storage
Retries during a degradation multiply load exactly when the system can least afford it, and synchronized retries create a wave that never lets the dependency recover. Backoff with jitter, a retry budget, and a breaker are the three-part fix.
Unbounded retention is the quieter one: every event, log line and version kept forever because deletion felt risky. Storage grows linearly with traffic and every query over it gets slower. Set retention from a requirement — 30 days for replay, 7 years for invoices — and tier older data to cheap storage instead of primary disks.
Worth remembering
- Ten sequential 20 ms calls is 200 ms of pure waiting; run independent calls in parallel or batch them.
- Fan-out to every follower at read time turns a celebrity post into a thundering herd.
- Over-fetching multiplies bytes at every layer: database, network, serialisation, client memory.
- Unbounded retention is a storage bill with no product requirement behind it.