Module 03 · Lesson 04

Load balancing without breaking sessions

Four algorithms, two layers, one question: what happens when a backend dies mid-request?

Intermediate 7 min read Updated 2026-11-01

What you will be able to do

Choose a balancing algorithm for uneven, stateful, and cache-heavy traffic
Explain the difference between L4 and L7 balancing
Design health checks that do not cause outages
Describe graceful draining during a deploy

Algorithms and when they lie

Algorithm	How it picks	Fails when
Round robin	Next backend in order	Requests have wildly different costs — a heavy query lands on an already busy box
Weighted round robin	Proportional to capacity	Capacity differs by more than the weights assume, or load is bursty
Least connections	Backend with the fewest in-flight requests	Requests are long-lived and uneven, or one call fans out internally
Least response time	Fastest recent latency	A cold backend looks great, then gets slammed
Consistent hashing	Same key to the same backend	A backend leaves and its share must be redistributed carefully

Round robin is fine for a fleet of identical stateless handlers serving similar requests. Least connections is better when request durations vary by an order of magnitude. Consistent hashing is what you choose when a backend holds a cache or a shard and a miss is expensive.

Power of two choices

Pick two backends at random and send the request to the one with fewer in-flight requests. It needs almost no coordination and avoids both the unlucky pile-up of pure random and the herd that least-connections can produce. Many production proxies do exactly this.

L4 versus L7

L4 balances TCP/UDP flows. It sees IPs and ports, forwards bytes, and is very fast, but it can only keep a connection on one backend.
L7 terminates the request, reads path, headers, and cookies, then opens a separate connection to a backend. It can route /api/* to one fleet and /media/* to another, retry a failed request, or apply per-route auth.
A common stack uses L4 at the network edge for raw throughput and L7 inside for routing — and you can say that in one sentence.

The practical difference: with L4, a long-lived WebSocket pins a client to one machine for its lifetime. With L7, each request can land anywhere, which is why keep-alive to backends and connection pooling matter for performance.

Health checks that help instead of hurt

Separate liveness from readiness

Liveness asks 'is this process broken?' and restarts it. Readiness asks 'should this instance receive traffic?' and removes it. Mixing them restarts healthy pods during a slow dependency.

Check dependencies shallowly

A readiness check should verify the instance can serve, not that every downstream is perfect. If it fails whenever the cache blips, you remove your whole fleet at once.

Require consecutive failures

Three failures in a row, not one, prevents a single slow probe from evicting a good node.

Drain before removing

Stop sending new requests, let in-flight ones finish, then deregister. Without draining, every deploy produces a burst of 502s.

The outage you cause yourself

If health checks are aggressive and every instance depends on the same database, a brief database stall marks every instance unhealthy simultaneously. The balancer then has nowhere to send traffic. Keep the check local, and use load shedding rather than mass eviction.

Sessions and state

Sticky sessions keep a user on one backend by hashing a cookie. They work, and they are a trap: they make scaling uneven, break on deploys, and turn one machine's failure into a wave of logged-out users. Prefer a shared session store or a signed token the client carries. If you must be sticky — a WebSocket, an in-memory game room — then be explicit about what happens when the node dies: reconnect, rejoin, replay from a cursor.

Bad:   user -> same pod forever (one pod dies = those users are down)
Better: user -> any pod, session in Redis (pod dies = nobody notices)
Needed: user -> socket pod, registry maps user->pod (pod dies = reconnect + replay)

Worth remembering

Round robin assumes every request costs the same; least connections assumes you can count; hashing assumes locality matters.
L4 routes packets, L7 reads the request — the second enables path routing, header rules, and per-route TLS.
Health checks that restart a slow pod can turn a latency problem into an outage.
Sticky sessions fix a symptom; shared state fixes the cause.

Checkpoint

Answer all 3 questions to finish the lesson.

1. Backends host a large in-memory cache. Which algorithm best preserves cache locality?

Hashing a key to a backend keeps repeated requests for the same data on the node that already has it in memory; random routing would scatter and miss.

2. Which is an L7 capability that L4 cannot provide?

L7 parses the request, so it can route on path or headers and retry at the request level. L4 only sees addresses and ports.

3. Why can a readiness check that calls the database remove an entire fleet at once?

Shared dependencies are a common cause of correlated failure. Readiness should reflect local ability to serve, with graceful degradation instead of mass eviction.

← PreviousHTTP that behaves well, and TLS that is not an afterthought Next →REST, GraphQL, gRPC: pick by shape of change