Module 12 · Lesson 07
Design a notification service
Multi-channel delivery where the hard parts are preferences, deduplication and not waking someone twice.
What you will be able to do
- Separate notification creation from channel delivery
- Resolve user preferences and quiet hours
- Deduplicate across retries and multiple events
- Handle provider limits and permanent failures
Requirements and estimates
- Functional: accept events (order shipped, password changed), decide channels, render templates, deliver, track status.
- Non-functional: p99 under 2 s for email, near-real-time push, no duplicate delivery, backpressure per provider.
- Scale: 1B notifications/day across four channels.
1B/day = ~11,500/s average, 50k/s peak per channel queues: push, email, sms, in-app dedup store: 1B ids x 7 days x ~100 B = ~70 GB (or a bloom filter)
Architecture
Ingest
Services publish domain events to one topic. The notification service owns interpretation, not the producer.
Resolve
Load preferences, quiet hours, locale and consent. Drop or defer; return without error.
Render
Template plus data per channel. Render at send time so links and times are correct.
Enqueue
One queue per channel and provider, so an SMS outage does not stall push.
Deliver and record
Worker calls the provider with an idempotency key, records status, retries transient errors.
| Channel | Typical limit | Retry policy | Dedup window |
|---|---|---|---|
| Push | Provider throughput cap | 3 attempts, 1-30 s backoff | 24 h |
| ~50-500/s per provider | 5 attempts, minutes | 7 days | |
| SMS | Per-country caps, expensive | 2 attempts, fail to fallback | 7 days |
| In-app | None, database write | Retry on DB error | 7 days |
Deduplication is not optional
Events retry, workers redeliver, and two upstream services may both emit "order shipped". Without a key of (notification id, channel, user), a customer gets the same email three times and support hears about it.
Quiet hours and preferences
Preferences are per user, per channel, per category: transactional messages always send, marketing respects opt-out, and quiet hours defer non-urgent push to the next morning. Store quiet hours in the user's timezone, not the server's. If a notification is deferred, it must be re-enqueued with a delay rather than dropped.
The full sentence
"One ingest topic, per-channel queues, preference and quiet-hour resolution before rendering, an idempotency key of notification id plus channel for deduplication, and a dead-letter queue for permanent provider failures."
Worth remembering
- One ingest topic, then a queue per channel, so a slow channel cannot block the others.
- A notification id plus channel is the deduplication key; retries must not create a second email.
- Preference checks belong before rendering, to avoid wasted work and to honour quiet hours.
- Permanent failures go to a dead-letter queue; transient ones retry with backoff and a cap.
Checkpoint
Answer all 3 questions to finish the lesson.
1. Why one queue per channel instead of a single delivery queue?
Isolating channels keeps an SMS provider outage from delaying push or email, and lets each be rate-limited independently.
2. What makes a good deduplication key?
It must identify one intended delivery. Bodies can change on re-render and timestamps differ across retries.
3. A user is in quiet hours and the message is a marketing push. What should happen?
Quiet hours are a timing constraint, not an opt-out. Deferring preserves the intent without violating the preference.