Module 12 · Lesson 05
Design a news feed
The classic fan-out trade-off, plus the part people forget: ranking and caching the result.
What you will be able to do
- Choose a fan-out strategy from follower distribution
- Design cursor pagination for a mergeable feed
- Place ranking and caching in the read path
- Handle deletes and edits across precomputed timelines
Requirements and estimates
- Functional: reverse-chronological or ranked feed of posts from followed accounts, cursor pagination, new posts appear on refresh.
- Non-functional: p99 under 200 ms for the first page, 200M daily users, roughly 50 feed loads per user per day.
- Scale: 500k feed reads/s peak, 5M posts/day.
reads: 200M users x 50 loads/day = 10B/day = ~115k/s avg, 500k/s peak
posts: 5M/day = ~60/s
fan-out: average 200 followers -> 1B timeline writes/day at 60/s posts
a 10M-follower account alone = 10M writesFan-out strategies
| Strategy | Write cost | Read cost | Breaks at |
|---|---|---|---|
| Push on write | Posts x followers | One list read | Celebrities |
| Pull on read | One post insert | Followees x recent posts | Users following thousands |
| Hybrid | Normal accounts push; large accounts pull | Merge a few sources | Complexity, cache invalidation |
The hybrid is standard: below a follower threshold (say 100k), push post ids into follower timeline lists in a stream job; above it, skip the push and merge that account's recent posts at read time. The threshold is a tuning number — say so rather than pretending it is exact.
Data model
timeline:{user_id}— Redis sorted set or list of post ids, capped at ~1,000 entries.posts— wide-column table keyed by author, sorted by time, holding content reference and counters.follows— adjacency table, read to determine who receives a push.feed_cache:{user_id}— rendered first page, TTL of a minute, invalidated on refresh.
Pagination must be stable
Offset pagination shifts when new posts arrive, so page two repeats items from page one. Use a cursor of (timestamp, post id) or (rank score, post id) and return everything strictly after it.
Read path
Check the rendered cache
Most first-page loads end here; TTL of 30-60 s is enough for a feed.
Gather candidates
Read the pushed timeline, then fetch recent posts from large followees, then merge and dedupe.
Hydrate
Batch-fetch post bodies, author profiles and counters with a multi-get, not one query per post.
Rank and paginate
Score with recency, affinity and engagement, take the page size plus a small buffer, return a cursor.
Deletes and edits are the operational tax of precomputation. A delete must remove the id from timelines lazily — filter at hydration and let TTLs expire the entry — or delete from a capped list, which is O(n) per follower. Lazy filtering plus a tombstone is the usual compromise.
The shape of a strong answer
"Hybrid fan-out: push for normal accounts, pull for large ones, capped at 1,000 ids per timeline. Read merges both, hydrates in batches, ranks per viewer, and paginates on (score, id). Deletes filter at hydration with a tombstone."
Worth remembering
- Push is cheap to read and expensive to write; pull is the reverse. Most large systems use both.
- A celebrity's posts should not be written to millions of timelines synchronously.
- Cursor pagination on (rank, id) survives new inserts; offsets do not.
- Precompute the candidate set, rank per viewer at read time — ranking is personal.
Checkpoint
Answer all 3 questions to finish the lesson.
1. An account with 10 million followers posts. Why is push fan-out unsuitable?
Write cost scales with follower count. Large accounts are pulled at read time and merged instead.
2. Which pagination approach is stable as new posts arrive?
A cursor anchors to a position in the ordering, so newly inserted items do not shift later pages.
3. Where should ranking happen?
Ranking depends on the viewer, so it cannot be fully precomputed per author. Ranking a few hundred candidates is cheap.