Why is it slow? A method, not a hunch
Measure the layer, then the hop, then the line of code. Guessing wastes the incident.
What you will be able to do
- Turn a vague slowness report into a measurable symptom
- Bisect a request path with traces and timings
- Identify N+1 queries and missing indexes from evidence
- Distinguish saturation from contention from a slow dependency
Start with a precise symptom
Before touching anything, write four facts: which endpoint, which percentile, which time window, and what the number was before. "Checkout is slow" becomes "checkout p99 went from 400 ms to 2.1 s starting at 14:05, on all instances". The second statement already narrows the search.
| Observation | Implies |
|---|---|
| All endpoints slow, all instances | Database, network or shared dependency |
| One endpoint slow | The code path or its query |
| One instance slow | Noisy neighbour, bad host, GC, disk |
| Slow at high traffic only | Queueing or contention, not raw work |
| Slow for large accounts only | Query shape, N+1, or fan-out |
| Slow after a deploy | New code path, migration, config |
Bisect the path
A trace turns five guesses into one. If the waterfall says 40 ms in the gateway and 1,900 ms inside the cart service, the answer is in that service — not the CDN, not DNS, not the load balancer. Repeated across a sample of slow requests, the pattern becomes obvious.
cart-service 1,900 ms db.query 1,820 ms (312 calls) <-- N+1 redis.get 12 ms serialize 8 ms
Three hundred and twelve queries for one request is the N+1 signature: fetch a list, then fetch something for each item. The fix is a single batched query. It is rarely a caching problem, and adding a cache to hide it also caches the inefficiency.
Saturation versus contention
- Saturation: CPU, connections, disk or network near its limit. Latency rises because work waits in a queue. Fix by adding capacity or reducing work.
- Contention: threads wait on a lock, a row, or a single hot key. CPU is idle but throughput is low. Fix by partitioning or shortening the critical section.
- Slow dependency: a downstream call takes a fixed long time. Fix with a timeout, caching, or async work — not with more application threads.
- Pathological input: one request triggers a scan or a huge payload. Fix with limits at the edge before it reaches the service.
Worth remembering
- "Slow" is not a symptom. p99 latency, on one route, at one time, compared to a baseline is.
- A trace waterfall finds the hop in one request; aggregates only tell you which service.
- Latency that grows with the number of rows is a query shape problem, almost always N+1 or a missing index.
- Saturation shows queueing; contention shows throughput collapsing while CPU is idle.