Module 10 · Lesson 03
Rate limits, quotas and the public edge
Every public endpoint will be called by something you did not design for.
What you will be able to do
- Place rate limits at the right layer
- Choose between token bucket, leaky bucket and sliding window
- Design quota keys that do not become a hot shard
- Name the validation and egress protections a public API needs
Algorithms
| Algorithm | Allows bursts | State | Use for |
|---|---|---|---|
| Token bucket | Yes, up to bucket size | Tokens + timestamp per key | APIs with bursty clients |
| Leaky bucket | No, smooth output | Queue depth | Shaping outbound traffic |
| Fixed window | Yes, at window edges | One counter per window | Cheap coarse limits |
| Sliding window log | Precise | Timestamp per request | Strict limits, small keys |
| Sliding window counter | Approximate | Two counters | Balanced default |
Fixed windows have a well-known flaw: 100 requests at 12:00:59 and 100 at 12:01:00 both pass a 100-per-minute limit, producing 200 in two seconds. Sliding-window counters fix this closely enough with two numbers.
allow(key):
now = clock()
refill = (now - bucket.ts) * rate
bucket.tokens = min(capacity, bucket.tokens + refill)
if bucket.tokens >= cost:
bucket.tokens -= cost; bucket.ts = now; return true
return false # with Retry-After = (cost - tokens) / rateAlways return Retry-After
A 429 without a hint produces retry storms from well-behaved clients. The header also makes the limit debuggable, which reduces support load.
Where to enforce
- Edge / CDN / WAF: coarse per-IP limits and bot rules. Cheap, blocks floods before they reach you, but IPs are shared behind NAT.
- API gateway: per-account and per-API-key limits with quota tiers. The usual primary enforcement point.
- Service: per-resource fairness — one tenant must not consume the whole pool. Also protects expensive endpoints.
- Downstream dependency: concurrency caps, not request-rate limits, because what matters there is in-flight work.
Layering is the point: an edge limit stops a flood, a gateway limit enforces the contract, a service limit enforces fairness. Skipping the service layer means one heavy tenant degrades everyone else even while under the global limit.
Redis INCR + EXPIRE
sub-ms
Use a single atomic script
Local counter accuracy
± instances
N instances can overshoot N×
Per-IP limit behind NAT
shared
An office or carrier shares one IP
Request body cap
1-10 MB
Enforce before parsing
The public edge checklist
Authenticate before expensive work
Reject unauthenticated requests at the gateway, before parsing bodies or touching the database.
Cap sizes and pagination
Maximum body, maximum page size, maximum upload. An unbounded page size is a free denial of service.
Validate shape, then content
Schema validation first, then business rules. Never build queries from raw input; parameterise.
Restrict outbound calls
If the server fetches user-supplied URLs, validate the scheme and block internal ranges to prevent server-side request forgery.
Log and alert on abuse patterns
Spikes in 401s, 429s and validation failures are the earliest signal of an active attempt.
Frame it as protection, not gatekeeping
"Limits are layered: per-IP at the edge for floods, per-key at the gateway for the contract, per-tenant concurrency in the service for fairness, all returning Retry-After. Body and page sizes are capped before parsing."
Worth remembering
- Token bucket allows bursts and a steady average, which matches how real clients behave.
- Limit at the edge by IP and at the service by account — one alone is trivial to bypass or too coarse.
- A distributed counter needs an atomic operation or a small allowed overshoot; per-node counters leak capacity.
- Validate size, type and shape before parsing, and cap response size for pagination.
Checkpoint
Answer all 3 questions to finish the lesson.
1. Which algorithm permits a burst up to a fixed size while enforcing a steady average?
Tokens accumulate up to capacity, so a client may spend a burst and then is limited to the refill rate.
2. Why is a per-IP limit alone insufficient?
IP is neither unique to a user nor stable for an attacker, so it is both too coarse and too easy to evade.
3. A server fetches image URLs supplied by users. What attack must be blocked?
User-controlled URLs let an attacker reach internal metadata endpoints. Validate schemes and block private address ranges.