Module 03 · Lesson 06

Protecting the API from its own users

Rate limits, quotas, backpressure, and graceful degradation — the difference between a spike and an outage.

Intermediate 7 min read Updated 2026-11-01

What you will be able to do

  • Choose between token bucket and sliding window rate limiting
  • Explain where limits belong in a multi-layer system
  • Design graceful degradation for an overloaded dependency
  • Describe how to find and stop abusive traffic

Two algorithms, two intents

Token bucket
A bucket refills at a steady rate; each request spends a token. Because the bucket holds a small surplus, a short burst is allowed. This is the right fit when users legitimately spike — a mobile app syncing on reconnect.
Sliding window
Count requests in the trailing window and reject past the threshold. Fairer and easier to reason about for billing, but a burst just over the edge is refused even if the long-run rate is fine.
Fixed window
The cheap version: a counter per minute, reset on the boundary. Two bursts either side of the boundary let through up to double the limit, which is usually acceptable for coarse protection.
Concurrency limit
Cap in-flight requests rather than rate. This is often the most direct protection for a slow downstream: it bounds the work, not just the arrival rate.

In a distributed fleet, the counter must be shared or the limit multiplies by the number of nodes. Redis with an atomic increment, or a local bucket with a small overshoot tolerance, are both defensible — say which and why.

INCR  ratelimit:{tenant}:{minute}
EXPIRE ratelimit:{tenant}:{minute} 60

if count > limit: 429 with Retry-After: seconds-to-window-end

Where the limits live

LayerLimitWhy here
Edge / WAFIP and coarse global capsStops floods before they consume your capacity
GatewayPer-key, per-tenant quotasIdentity is known, and one place to change policy
ServicePer-endpoint cost limits, concurrency capsExpensive endpoints need their own budget
DatastoreConnection pool sizes and query timeoutsThe last line of defence for shared resources
Cost, not count
A search request that scans a million rows is not equal to a lookup by id. Weighted limits — where a heavy endpoint costs more tokens — protect capacity much better than counting requests equally.

Overload: three honest answers

  • Queue it — good when work is deferrable and the user is not waiting. A video transcode queue can absorb a spike.
  • Shed it — refuse the excess with 429/503 quickly. Cheap, honest, and better than accepting work you cannot finish.
  • Degrade it — serve a cheaper version: cached results, fewer recommendations, a static fallback. Users get something instead of an error.

The failure mode to avoid is accepting everything and finishing nothing. Latency climbs, queues grow, retries multiply, and a system that could have served 80% of traffic at full speed serves 0% at ten seconds per request.

Instead
Circuit breaker open forever after one failure
Do this
Half-open probes at intervals so recovery is automatic
Instead
Retry storm from every layer
Do this
One retry layer, jittered backoff, and a budget shared across the request
Instead
429 with no body or headers
Do this
Retry-After plus a machine-readable reason so clients can behave
Instead
Limits only in the service, after auth and parsing
Do this
A cheap edge rule too — auth and JSON parsing cost CPU you wanted to protect

Finding the abusers

You cannot limit what you cannot attribute. Emit a metric per key, tenant, endpoint, and status code, then watch the top talkers. Most incidents come from a small number of clients: a retry loop in someone's batch job, a scraper with no rate control, or your own mobile app on an old version hammering a deprecated endpoint. Look at the shape of the traffic before writing a rule — a limit applied to the wrong dimension just breaks honest users.

Worth remembering

  • Token bucket allows bursts up to the bucket size; sliding window is fair but stricter. Pick by product intent.
  • Limits belong at the edge for cheap, global protection and in the service for per-tenant fairness.
  • Queueing, shedding, and degradation are three different answers to overload — say which you are doing and why.
  • Always tell the client when you limit them (429 + Retry-After), or they will retry harder.

Checkpoint

Answer all 3 questions to finish the lesson.
1. Users legitimately burst traffic after reconnecting from offline mode. Which limiter fits best?
Token bucket refills steadily and lets a small surplus through, absorbing a legitimate burst while still bounding the long-run rate.
2. Why put a coarse IP limit at the edge as well as per-tenant quotas at the gateway?
Rejecting at the edge costs almost nothing; the same request rejected later has already consumed TLS, parsing, and auth work you were trying to protect.
3. What is the main risk of accepting all requests during overload instead of shedding?
Queueing delays compound with retries, so total goodput collapses. Fast, honest rejection preserves capacity for the traffic that matters.
← PreviousREST, GraphQL, gRPC: pick by shape of change