Module 05 · Lesson 01

Where a cache actually helps

A cache is a bet that the same data will be asked for twice, soon, and can be slightly stale.

Intermediate 7 min read Updated 2026-11-01

What you will be able to do

  • Estimate the capacity gain from a given hit ratio
  • Place a cache at the right layer of a design
  • Explain what must be true for a cache to be safe
  • Recognise when a cache is the wrong fix

The arithmetic of a hit ratio

If 100,000 reads per second arrive and the hit ratio is 95%, 5,000 reach the database. At 99%, 1,000 do. This is why a small improvement in hit ratio is worth more than an extra database replica, and why the hit ratio belongs on a dashboard.

reads/sec        = 100,000
hit ratio        = 0.95
db reads/sec     = 100,000 x (1 - 0.95) = 5,000

hit ratio        = 0.99
db reads/sec     = 1,000   -> one replica you no longer need
In-process memory read
~100 ns
Per service, no network
Redis round trip
0.5-1 ms
Same datacentre
SSD read
0.1-0.2 ms
Plus query planning on top
Cross-region cache
70+ ms
Usually worse than a local query

Note the third row. A cache in the same building saves network and query work, but a database with the row in its buffer pool is already sub-millisecond. The cache wins when it avoids expensive computation, serialisation of large objects, or a fan-out — not merely because it is memory.

Layers, and what each one protects

LayerHoldsProtects
Browser / clientStatic assets, API responsesYour bandwidth and the user's latency
CDN edgeImages, video, JS, cacheable JSONYour origin entirely for cache hits
API gatewayAssembled responsesBackend CPU for repeated identical requests
Service memoryHot objects, config, computed viewsNetwork hop and deserialisation
Shared cache (Redis)Sessions, counters, query resultsDatabase reads across all instances
Database buffer poolPages recently readDisk I/O

Each layer multiplies the effect and adds a place where staleness can hide. In a design round, name two or three layers with a reason, not six because they exist.

The three questions to answer out loud
"What is being cached, how stale may it be, and who invalidates it?" If you cannot answer all three, the cache is a bug with a fast response time.

When not to cache

  • Low hit ratio. A long tail of unique requests fills memory with one-hit objects and evicts the hot ones.
  • Writes dominate. Every write invalidates, so a write-heavy path gets little benefit and extra complexity.
  • Correctness is the product. Balances, inventory, and permissions should be read from a source that is authoritative, or cached with a version the reader verifies.
  • The query is the problem. An unindexed scan cached for 60 seconds is still a scan sixty times an hour.

One more: a cache hides load from capacity planning. Teams discover their real traffic only when the cache is cold after a deploy — the thundering herd. Mitigate with request coalescing (one loader per key), staggered warm-up, and a TTL jitter so entries do not all expire together.

Measure before adding
Know the read rate, the query cost, and the ratio of distinct keys to requests.
Choose the smallest scope
A per-instance cache with a short TTL may be enough and needs no shared infrastructure.
Bound it
Set a memory limit and an eviction policy; an unbounded cache is an outage waiting for a memory limit.
Make misses cheap
Coalesce concurrent misses, and protect the database with a timeout so a cold cache does not become a stampede.

Worth remembering

  • Cache hit ratio determines everything: 90% means one in ten requests reaches the database, 99% means one in a hundred.
  • The working set — the hot subset actually read — is usually far smaller than total data.
  • A cache is safe only when stale data is acceptable and invalidation has an owner.
  • Adding a cache to a slow query hides the problem; fixing the query is still the cheaper win.

Checkpoint

Answer all 3 questions to finish the lesson.
1. Reads are 100k/sec and the cache hit ratio is 95%. How many reads reach the database?
100,000 × (1 − 0.95) = 5,000. The miss rate, not the hit rate, is what sizes the backing store.
2. Which condition makes a cache unsafe?
Caching assumes bounded staleness with a defined invalidation path. Without both, the cache silently becomes a second, wrong source of truth.
3. After a deploy clears the cache, the database is overwhelmed. What is the standard mitigation?
A cold cache turns every request into a miss at once. Coalescing collapses duplicate loads and staggering prevents a synchronized stampede.
← PreviousSearch engines and analytics stores