Caching and Routing Change the Request Path

A cache is not a performance feature bolted onto the side of a system — it rewrites the request path. It decides who answers a request (a process-local map, a Redis node, an edge PoP, the origin), with what data (the current row or a copy from up to N seconds ago), and at what latency. Routing decides which of your servers gets the request in the first place. Together they control freshness, locality, load distribution, and — when they are wrong — the specific class of bug where a user sees someone else's data.

The economics are why nobody skips this. A same-AZ Redis read is roughly 0.5–1 ms; a primary database read is 1–10 ms; a cache miss that fans out to a join across three tables can be 50–200 ms. Serving 10,000 requests per second with a 97% hit rate means only 300 requests per second touch the origin — the cache is the difference between an origin sized for 300 rps and one sized for 10,000.

Cache-aside, step by step

In cache-aside (lazy population) the application owns the logic: check the cache, load from the source on a miss, write the result back with an expiry.

async function getUser(id) {
  const key = `user:${id}`;
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit);
 
  const row = await db.query("SELECT * FROM users WHERE id = $1", [id]);
  await redis.set(key, JSON.stringify(row), { EX: 60 }); // TTL in seconds
  return row;
}

Read-through is the same flow with the cache in front of the data access layer; write-through writes the cache and the database together; write-behind writes the cache first and flushes later. The trade-off is always the same axis: how stale may this answer be, and what happens when the write and the fill race.

That race is the reason a bare cache-aside implementation is incomplete:

1. reader A: cache miss, reads row (total = 100)
2. writer B: updates row (total = 120), invalidates cache
3. reader A: writes its stale copy back with TTL 60s
   -> every reader for the next 60s sees total = 100

The fix set: invalidate on write after the commit, keep the TTL short enough that a stale write self-heals, or version the key (user:42:v17) so an old fill can never overwrite a newer one.

Keys: everything that changes the answer

A cache key must include every input that changes the result — the key is the correctness argument. Tenant, locale, currency, authorization scope, API version, feature-flag state, and query parameter order all belong in it when they affect the response:

const key = [
  "products:list",
  `tenant:${ctx.tenantId}`,   // omit this and one tenant sees another's catalog
  `locale:${ctx.locale}`,
  `plan:${ctx.plan}`,         // pricing differs by plan
  `v:${SCHEMA_VERSION}`,
  stableStringify(filters),   // sort keys so ?b=1&a=2 and ?a=2&b=1 agree
].join("|");

The production horror stories are almost always an omitted component: a shared CDN cache serving a personalized page, or a per-tenant list keyed only by the query. Note also that Authorization and Cookie headers do not participate in most caches' keys unless you explicitly configure Vary — assume anonymous caching unless proven otherwise.

Freshness, expiry, and the miss storm

TTL-only freshness is simple and usually right; explicit invalidation is immediate but distributed. What both must survive is the expiry cliff: at 10,000 rps, a hot key's TTL expiring means up to 10,000 requests can arrive in the same second and all find nothing — each one a synchronous query to the origin. The standard defenses:

• Single-flight (request coalescing): one in-flight load per key; the other callers wait for it. Redis SET NX with a short lock, or an in-process promise map.
• Early revalidation: refresh a key at 80% of its TTL instead of at expiry.
• Jitter the TTL: 60s ± 10% so a million keys written at once do not expire together.

Routing: which server answers

Load balancing decides distribution. Round-robin spreads connections evenly but knows nothing about connection cost or request price; least-connections adapts to heterogeneous requests; IP hash is stable but skews when a client NAT pool collapses dozens of users onto one address. Every layer also has its own health check, and health is a timeout: a backend that answers slowly gets ejected only after the checker's own probe exceeds its limit, so there is always a window where the balancer still routes to a dying server.

Sticky routing (by cookie or client IP) improves locality — sessions and caches stay warm — but concentrates load during failover and makes capacity uneven. Consistent hashing is the usual compromise for cache nodes: when a node joins or leaves, only that node's share of keys moves (roughly 1/N with the right ring design), instead of remapping the whole keyspace as modulo hashing does. See consistent hashing and distribution and how a load balancer distributes requests.

Where this bites in production

• Stale data after a user action is the top cache bug: fix ordering (commit, then invalidate, then respond), not TTL length alone.
• Cache keys missing authorization context turn into data-leak incidents, not latency incidents.
• A cache that works at 3 rps in staging fails at 10,000 rps in production — the miss storm only exists at scale. Load-test the cold start, not only the warm steady state.
• Measure what you actually serve: log hit/miss per layer and alert on hit rate dropping, because a slow decline in hit rate is an origin outage in progress.

Further reading: the AWS Well-Architected Framework.