Caching Layers and TTL Tradeoffs

Caching is the least controversial latency bet in systems design: a Redis GET takes 100-200 microseconds on a warm box, a local Postgres point lookup 1-5 milliseconds, and a remote database over the network 5-20 milliseconds. A single cache hop buys you a 10-100x read speedup. But a cache only helps if its contents are correct enough — and "correct enough" is a contract you design, not an accident.

Cache-aside: the default pattern

Cache-aside (lazy population) is what most services implement without naming it:

READ:  cache lookup → hit? return
       miss → query DB → write cache with TTL → return
WRITE: write DB first → delete cache key(s)

Two details matter. First, the cache is populated lazily, so the first request for any key pays the full DB latency — and in production, first requests cluster: a day's hot keys get fetched around the same time, and warm cache entries all get written at nearly the same moment. Second, invalidation must happen after the DB write, and it should be a delete, not an update. Updating the cache with the new value races with concurrent writers; deleting forces the next reader to fetch a consistent snapshot. A delete leaves a tiny stale-read window between the DB commit and the delete; an update race leaves stale data with no expiration at all.

Write-through: trading write latency for read consistency

Write-through updates the cache and the database in the same write path, synchronously. Every write now pays two round trips (cache + DB), so a 1ms write becomes 2-4ms. In exchange, hot keys are always fresh — no invalidation race, no miss-then-fetch on the read side. This suits workloads that rewrite the same keys (session state, counters) where cache-aside's lazy repopulation would hammer the DB. The catch: write-through still needs a TTL, because a cache that never expires grows without bound and can serve state the DB later rolled back.

TTL vs invalidation: the real tradeoff

Invalidation is precise but requires knowing every key a mutation touches. That works for a single normalized row; it fails for aggregates — cached counts, feed pages, denormalized views — where one write affects dozens of keys. TTLs are the honest answer to aggregation: they bound staleness (a key is at most ttl seconds stale) without bookkeeping. The price: some fraction of reads are stale by design, and expired keys create refill traffic you cannot see coming.

Set TTLs with arithmetic: a 60-second TTL on a key fetched 50 times per second means ~50 cache writes per minute per key, plus the occasional DB refill. A TTL is a staleness budget and a write budget — cutting it from 60s to 10s buys freshness at 6x the refill load.

Cache stampedes: when expiry synchronizes

The failure mode that actually melts databases is the stampede. One hot key (the viral post, the shared config) expires; every concurrent request misses simultaneously; they all refill at once; the DB gets N× the traffic it would have received — N being the number of concurrent requests. With a 10ms DB query and 1,000 QPS on one key, that's 10 requests in flight; at 100k QPS and a 100ms query, a stampede is 10,000 concurrent DB hits.

The standard mitigations, in increasing order of sophistication:

• TTL jitter: add 0-10% random variance to every expiry so keys don't die in lockstep.
• Request coalescing (single-flight): only one in-flight refill per key; everyone else waits on its result.
• Stale-while-revalidate: serve the stale value while the refill runs in the background — TTL becomes a lower bound on freshness instead of a death sentence.
• Probabilistic early expiry: refill at a fraction of the TTL with probability proportional to age — smooths refill load without serving expired data.

get(key):
  v = cache.get(key)
  if v: return v
  lock = acquire(key_lock)          # only ONE refill per key
  if lock.acquired:
    v = db.query(key)
    cache.set(key, v, ttl + jitter)
    release(key_lock)
    return v
  return wait_on(lock)              # everyone else waits, no stampede

Miss storms: the cache as a single point of failure

A stampede is a single key. A miss storm is every key: cache restart, OOM eviction of the whole keyspace, a network partition, a deploy-time flush. Suddenly 100% of reads go to the database at full production QPS — and the DB was sized for the ~5-10% miss rate the cache guaranteed, not 100%. The DB saturates, timeouts compound, and every retry is another miss, which keeps the storm alive.

This is why real systems treat the cache as a liability to bound, not a guarantee: keep the DB's headroom sufficient for a cold start, warm the cache before enabling traffic (a Kubernetes startupProbe plus a warmup script), and size the DB for the miss rate you can survive, not the one you hope for.

The summary is uncomfortable: invalidation makes caches correct, TTLs make them manageable, and jitter plus single-flight make them survivable. You need all three — each protects against a different way the cache can lie.