Module 08 · Lesson 03
Scaling a fleet without manual surgery
The interesting part is not adding servers. It is what breaks when the number of servers changes.
What you will be able to do
- Pick an autoscaling signal that reflects user pain
- Explain connection pool multiplication
- Remove state that blocks horizontal scaling
- Sequence scaling events so they do not oscillate
What signal
| Signal | Good for | Failure mode |
|---|---|---|
| CPU utilisation | CPU-bound workers | Latency rises long before CPU saturates on I/O work |
| Requests in flight | Web services | Needs a per-instance target you can justify |
| Queue depth | Workers | Depth alone ignores arrival rate |
| p99 latency | User-facing | Noisy; scale up late and down too early |
| Scheduled capacity | Predictable peaks | Cannot absorb a surprise |
A practical combination is scheduled baseline capacity plus concurrency-based autoscaling, with a fast scale-up policy and a slow, cautious scale-down. Being one instance too many for ten minutes costs cents; being one too few for one minute costs users.
The connection pool multiplier
Each instance with a pool of 20 connections is fine at five instances. At a hundred instances it is two thousand connections, and the database's per-connection memory turns into swap. This is why production systems put a pooler — PgBouncer, a proxy, or a serverless connection proxy — between services and the database, and why a design should state the expected connection count.
Postgres connections
~100-500
Practical limit, not theoretical
Per-connection memory
5-10 MB
Multiply by connection count
Pool per instance
5-20
Set from database capacity divided by instances
Scale-up lag
30-90 s
Scheduling, image pull, readiness
Autoscaling can amplify an outage
If latency rises because the database is saturated, adding instances adds connections and load. Cap the fleet size and make the database the known bottleneck in the design.
State that blocks scaling
- Sessions in memory. Move to a signed cookie or a shared store; otherwise every request needs its original instance.
- Local caches as correctness. Per-instance caches are fine for performance, wrong for permissions.
- Local file uploads. Any instance must be able to serve any file, so files go to object storage.
- In-process cron. Five instances means five executions; use conditional writes or a single-flight lease.
- Sticky websockets with no plan. Connections are fine, but reconnection storms after a deploy need backoff and jitter.
Make instances disposable
Any instance can serve any request; killing one is not an event.
Scale the stateless tier on concurrency
Target 60-70% utilisation so bursts have headroom.
Protect the data tier explicitly
Cap pools, add read replicas or caching, and know the connection ceiling.
Drain before termination
Remove from the load balancer, finish in-flight requests, then stop. Otherwise every scale-down drops requests.
The sentence that closes the loop
"The service tier is stateless and autoscales on in-flight requests with a cap. Sessions live in Redis. Database connections are pooled centrally, and scale-down drains connections before termination."
Worth remembering
- Scaling on CPU lags; queue depth or concurrency reflects load before latency degrades.
- Every instance opens its own database connections — 50 instances × 20 connections is a thousand connections.
- Sticky sessions and local caches are the two most common blockers to horizontal scale.
- Scaling is not instant: assume a minute or more before new capacity serves traffic.
Checkpoint
Answer all 3 questions to finish the lesson.
1. Fifty instances each hold a pool of 20 connections. What breaks?
Pools multiply with instance count. The database, not the service, sets the ceiling.
2. Why is CPU a poor autoscaling signal for an I/O-bound API?
I/O wait does not consume CPU. Concurrency or queue depth reflects the pressure that hurts users.
3. What must happen before an instance terminates during scale-down?
Without draining, in-flight requests are cut off and users see errors on every scale-down.