Module 06 · Lesson 03

Lag, backpressure and the queue that grows forever

An asynchronous system does not remove load, it stores it. The queue depth is the bill.

Intermediate 7 min read Updated 2026-11-02

What you will be able to do

Compute time-to-drain from arrival rate, service rate and backlog
Choose a backpressure strategy for a given stage
Distinguish a burst from a capacity deficit
Explain why an unbounded queue is an outage in slow motion

Little's law, used out loud

The average number of items in a system equals arrival rate times time spent in the system. For a queue that means backlog ≈ arrival rate × wait. Two consequences matter: a stable backlog with growing wait means throughput is falling, and a growing backlog means arrival is beating service.

arrival   = 2,000 events/s
service   = 2,400 events/s
backlog   = 1,200,000 events
drain     = 1,200,000 / (2,400 - 2,000) = 3,000 s = 50 min

if service drops to 1,900 events/s:
  backlog grows by 100 events/s, never drains

Fifty minutes of drain at current rates — and the moment service dips below arrival, the arithmetic stops producing a number. That is the shape of every consumer-lag incident: healthy for weeks, then a deploy, a retry storm, or a hot partition tips it over.

What lag actually measures

Metric	Meaning	Why it misleads
Lag count	Messages between newest and consumer offset	Meaningless without rate; 10k can be seconds or days
Lag age	Timestamp of newest message minus last processed	The one to alert on
Processing latency	Time per message inside the handler	Hides queueing time before the handler
Redelivery rate	Share of messages delivered more than once	Rises before lag does when a dependency is failing

Alert on the derivative too

Lag age tells you how stale the consumer is. The slope tells you whether it will recover on its own. A flat lag at two minutes is fine; a lag rising one second per second never recovers without intervention.

Four ways to push back

Slow the producer

Bound in-flight requests and block or reject at the edge. Cleanest when the producer is yours and the user can wait.

Drop with a policy

Sample telemetry, expire stale updates, coalesce to the latest state per key. Only valid where losing a message is acceptable — decide that explicitly.

Degrade the work

Serve a cheaper result: skip thumbnails, batch writes, answer from cache. Keep the queue moving at lower fidelity.

Refuse

Return 429 or 503 with retry-after. Honest refusal beats accepting work you will finish after it is useless.

There is a fifth option people reach for first: add consumers. It works only if partitions exceed consumers. With twelve partitions and twelve consumers, the thirteenth consumer idles, no matter how much CPU the machine has.

Kafka partition ceiling

~one consumer per partition

Adding consumers past this does nothing

Prefetch too high

100-500 msgs/worker

Turns one slow handler into a stall

Retry with backoff

3-5 attempts

Unbounded retries turn a blip into a backlog

Poison-message threshold

3-5 deliveries

Then dead-letter it; a blocker retries forever

Hot partitions break the average

Aggregate lag can be zero while one partition is hours behind. A single large customer or a popular product id concentrates traffic on one key. You see it only if you graph lag per partition. Fixes: split the hot key with a suffix and merge downstream, move that entity to a dedicated topic, or accept the limit and scale the handler for that key path.

Say this before you add workers

"Consumers scale with partitions, so first I'd check the partition count and per-partition lag. If one key is hot, more consumers won't help — I'd need to spread that key or isolate it."

Worth remembering

Backlog divided by (service rate − arrival rate) is the drain time; when arrival exceeds service, that number is infinite.
A queue converts a latency problem into a memory and money problem.
Backpressure is a product decision: drop, delay, degrade, or refuse.
Alert on lag age, not just lag count — a thousand messages behind in a fast stream is seconds, in a slow one it is hours.

Checkpoint

Answer all 3 questions to finish the lesson.

1. Arrival is 2,000/s, service is 2,400/s, backlog is 1.2M. Roughly how long to drain?

1,200,000 / 400 = 3,000 seconds, about 50 minutes. Drain time depends on the surplus, not on the backlog alone.

2. Which metric should trigger an alert?

Lag age converts an offset gap into user-visible staleness. A count means nothing without knowing the stream's rate.

3. A topic has 8 partitions with 8 active consumers and lag keeps rising. What is the first thing to check?

Consumers cannot exceed partitions, so adding one just idles. The remaining explanation is uneven load or slow handling, which per-partition lag exposes.

← PreviousDelivery guarantees, honestly stated Next →Event patterns that survive contact with production