The news feed must assemble, for each user, a ranked list of recent posts from people they follow, at 100M DAU scale. The decision that defines the whole architecture is fanout: when a user posts, does the machine push the post into every follower's feed, or does each follower pull and merge on refresh?
Push (fanout-on-write): the poster's post is written to each follower's precomputed timeline cache. Reads become a single cache fetch — 8.3M feed requests/sec at 5 refreshes/min × 100M DAU, each one a Redis read of a list of post ids plus a rank pass. Writes are expensive: a celebrity with 10M followers costs 10M cache writes per post.
Pull (fanout-on-read): the machine merges the poster's recent posts from each followee's wall at read time — 500+ followees × per-person queries per refresh, then rank and trim. Reads become the bottleneck, and merging 500 streams at 8M refreshes/sec does not fit in one box.
Hybrid is what ships: push for normal users, pull for celebrities above a follower threshold (e.g. 10K). The machine keeps per-user timeline caches in Redis, per-poster walls in a KV store, and a follow graph (social graph) in a graph DB or MySQL adjacency table.
Ranking: the machine scores posts by recency, affinity (how often the user engages with the poster), and post quality, then trims to ~500. Ranking happens at serve time on the cached id list — the cache holds ids, not rendered posts; post content is fetched in a batch GET from a post cache.
Data model: timeline:{user_id} → sorted post ids, wall:{user_id} → post ids, follows(follower, followee). Bottlenecks: fanout storms on big posters (queue the fanout work in Kafka and batch Redis writes); cold timelines for new users (lazy build on first refresh); cache invalidation on unfollow.
Post → API → fanout worker (Kafka)
├─ ≤10K followers: push into each timeline:{id}
└─ celebrity: skip; pull merges wall at read
Feed request → LB → feed service → Redis timeline → rank → batch post GET → render