Module 02 · Lesson 02

Latency, throughput and bandwidth

Three numbers that get confused constantly. Percentiles are the only honest latency.

Beginner 7 min read Updated 2026-11-01

What you will be able to do

  • Define latency, throughput and bandwidth without mixing them up
  • Explain why an average latency figure hides the problem
  • Describe the latency-throughput trade-off and when to take it
  • Read a p99 against a user-facing budget
TermQuestion it answersExample
LatencyHow long does one request take?p99 of 180 ms for a feed read
ThroughputHow many requests finish per second?5,000 reads per second
BandwidthHow many bytes move per second?2 GB/s outbound at peak
ConcurrencyHow many are in flight at once?1,000 open connections

Why percentiles, not averages

Averages blend the fast majority with the slow minority and produce a number no user experiences. If 99 requests take 10 ms and one takes 2 seconds, the average is about 30 ms — which describes nobody. The user who hit the slow request waited two seconds, and they are the user who leaves.

Say p50, p95 and p99 instead. p50 tells you the typical experience, p99 tells you what the unlucky users get, and the gap between them tells you whether the system has a tail problem: garbage collection pauses, lock contention, cold caches, or one hot shard.

Little's Law, in one line
Concurrency = throughput x latency. At 2,000 requests per second with a 200 ms latency, 400 requests are in flight at any moment. If your connection pool holds 100, the queue grows until timeouts — and this is arithmetic, not opinion.

The trade-off you are allowed to make

Batching raises throughput by amortising fixed cost per item, and it costs latency because the first item waits for the batch to fill. Both directions are legitimate; the mistake is not choosing.

Take latency when a human is waiting
Page loads, typing suggestions, button presses. Prefer small requests, parallel calls and caches over batching.
Take throughput when nobody is waiting
Analytics, transcoding, indexing, email. Batch ten thousand events into one write and cut the per-item cost by an order of magnitude.
State the trade when you make it
"I would batch click events for up to one second, which costs a little freshness on the analytics dashboard and cuts write volume by 100x." That sentence is the whole answer.
Q. The dashboard shows p50 latency is fine but users complain. What now?
Look at p99 and the slowest endpoints, not the aggregate. A healthy median with a bad tail usually means a subset of requests is slow: one hot shard, a cache-miss path, a timeout that is being waited out, or a retry storm. Slice latency by endpoint and by shard before changing anything.

Worth remembering

  • Latency is one request's duration; throughput is how many complete per second; bandwidth is bytes per second.
  • Averages lie. A p50 of 20 ms with a p99 of 2 s is a system that is slow for one user in a hundred, all day.
  • Batching and queueing raise throughput and hurt latency. Choose deliberately per workload.
  • Little's Law: concurrency equals throughput times latency. It predicts queue growth before you see it.

Checkpoint

Answer all 2 questions to finish the lesson.
1. Which figure best describes what a typical user experiences?
The median, p50, is the middle request. The mean is dragged around by outliers and describes no actual user.
2. A nightly indexing job switches to batches of 10,000 records. What is the effect?
Batching amortises per-write overhead, so more records complete per second, but each individual record waits for its batch. Fine for a batch job, wrong for a page load.
← PreviousScalability