The distributed cache must serve a 10TB working set at 1M QPS with sub-millisecond reads and 99.9% availability. It sits between stateless API servers and the database, turning 1M QPS into ~10K QPS of DB traffic at a 99% hit rate.
Placement uses consistent hashing: keys hash onto a ring, each node owns the arc to its successor, and each key's replica lives on the following node. A node failure rehashes only 1/N of keys (here ~1/320) onto neighbors, unlike a naive hash mod which would invalidate everything. The machine handles node churn with virtual nodes to keep ownership balanced when nodes differ in capacity.
Eviction is LRU per node, plus TTLs set at write time. A key that falls off the tail and a key that expires look identical to clients — both trigger a refill from the DB. Refill is where caches die: a cold key hit by 100K simultaneous requests means 100K identical DB queries. The machine mitigates with single-flight (one in-flight refill per key; the rest wait), TTL jitter (±10%) to desynchronize expiry waves, and a pre-warm path for known hot keys.
Hot keys are the second killer: one key doing 200K QPS pins one node. The machine splits hot keys with key-suffix sharding — key:0..15 — at the client library level, and keeps a small L1 in-process cache in each API server for the hottest keys.
Data model is a simple KV: key → (value, ttl, version). Version gates stale-write — a slow refill must not overwrite a newer value written directly to the DB. The consistency contract is eventual: the machine never promises the DB and cache agree at any instant, only that refills converge.
Bottlenecks: bandwidth on a single node for a hot key; full-node eviction cascades on failure (mitigate with replica promotion); and write-through vs write-back — the machine uses write-through for durability, accepting ~2x write latency.
Client lib (L1) → cache LB → ring of Redis nodes (consistent hash)
key on owner node; replica on successor
miss → single-flight refill → DB → populate with TTL(+jitter)