Module 02 · Lesson 10
Design principles that survive scrutiny
The defaults that make a design defensible when the interviewer pushes.
What you will be able to do
- Apply the simplicity-first test to a proposed component
- Decide when work moves out of the request path
- Design an operation to be safely retryable
- Use backpressure and observability as design elements, not afterthoughts
- Simplest thing that meets the requirement. Add a cache, a queue or a new database only when a number demands it. A three-box design that survives questions beats a nine-box design nobody can explain.
- Keep slow work out of the request path. Email, thumbnails, fan-out, indexing and third-party calls go behind a queue with retries. The user waits only for what they asked for.
- Make writes idempotent. An idempotency key on creation endpoints means a retried request returns the original result instead of duplicating a charge, a post or a message.
- One writer per piece of state. A single owner for a record keeps ordering simple. When you must have multi-writer, say how conflicts resolve.
- Push the cache toward the change. Cache close to where reads happen, and invalidate on write. TTL-only caching is a quiet data-freshness bug with a slow fuse.
- Apply backpressure. Bounded queues and rate limits turn overload into delay or rejection. Unbounded queues turn overload into an outage that starts long after the traffic did.
- Design the signal. Latency percentiles, error rate, queue depth and a health endpoint per dependency — decide at design time how you will know it broke.
- Price the design. Storage growth, egress and per-request compute are real. Saying "about 6 PB a year, so lifecycle to cold storage after 90 days" is senior behaviour.
How they show up in a round
| Principle | What it sounds like |
|---|---|
| Simplicity | "The requirement is 1,000 writes a second; one primary covers it, so I will not shard yet." |
| Async | "Thumbnailing is not needed to return the upload URL, so the client gets 202 and a worker does the work." |
| Idempotency | "The client sends a request key; a retry returns the first result instead of charging twice." |
| Single writer | "All writes for a conversation go to one partition, so ordering is free." |
| Backpressure | "The queue is bounded; at the limit we return 429 and let the client back off." |
| Cost | "We keep 30 days hot and move the rest to object storage; that caps the growth curve." |
The one that fails most often
Unbounded queues. They look like a smooth system: producers never block, the dashboard is green, and the backlog grows silently until a user asks why their upload from yesterday is still processing. A queue needs a length limit, an age alert, and a stated behaviour when full.
Q. Which principle should I apply first when I am stuck?
Simplicity. Ask what the smallest design is that satisfies every requirement you wrote down. Most stuck moments are actually over-design: the candidate is trying to justify a component they added for credibility rather than for a requirement.
Q. How mature should the observability answer be?
Three sentences: the four signals you would watch (latency p99, error rate, saturation, queue depth), where they live, and what alerts a human. Naming a specific failure you would catch — replication lag, cache hit ratio collapse, consumer lag — proves the answer is real.
Worth remembering
- Every component must earn its place by a requirement. 'It scales better' is not a requirement unless you have the number.
- Anything slow or unreliable that the user does not need immediately belongs behind a queue.
- Idempotency is what makes retries safe; without it, faults become data corruption.
- One writer per piece of state removes a whole class of consistency problems.
- Design the failure signal: if you cannot tell it is broken, it is not operable.
Checkpoint
Answer all 2 questions to finish the lesson.
1. Which operation must be idempotent before you allow automatic retries?
A retried non-idempotent write duplicates the effect — two charges. Reads are naturally safe to retry; writes need a key that makes the second attempt a no-op.
2. Why is an unbounded queue a dangerous design?
Producers never block, so nothing signals the imbalance until latency is measured in hours. Bounded queues with a defined full-behaviour convert overload into a visible, recoverable condition.