A Load Balancer Is Not a Magic Box

Every architecture diagram has a little box between "internet" and "servers" labeled LB. The box hides more complexity than any other single symbol in those diagrams — and almost every launch-day outage has a story that begins "we assumed the load balancer would handle it."

First, a genuinely hard question

You can't answer "which backend should handle this request" without a definition of "should".

That sentence is the load balancer's whole job description, in every dialect:

round robin .............. fairness by rotation — ignores everything else
least connections ....... fairness by in-flight work — needs good signal
consistent hashing ...... stability by key — (user id, tenant, IP) ↦ same backend
weighted ................ capacity as config — the fleet is not uniform

Each policy optimizes a different definition of "fair", and each fails on a different axis. Round robin is famously not slow-backend safe: a backend that takes 10× longer will keep receiving its share and keep being slow. Least connections fixes that but only as good as the health signal that feeds it — and the health signal is another request.

Layer 4 vs layer 7: the two jobs

The most important split is what the balancer sees:

Layer 4 (transport). The balancer looks at TCP/UDP headers and forwards flows, keeping the connection untouched end-to-end — it does NAT-like session-table tracking (i.e., it writes entries that look like client→VIP→backend) with a per-flow state table. Pros: it sees encrypted traffic without decrypting it, forwards ~line rate, and is ruthlessly generic. Cons: it cannot act on the content — no path-based routing, no health-aware content decisions, no caching; backend choice happens per connection, which limits stickiness granularity.

Layer 7 (application). The balancer terminates the connection, decrypts TLS, reads the HTTP request, and re-establishes an upstream connection. Pros: full URL/host/cookie-aware routing, can rewrite, can cache, can serve as an inspection point. Cons: it now has the cost of a full TLS termination per connection, a much larger session state, and — the part everyone forgets — a second set of connections to manage under failure. A layer-7 balancer's own connection pools to backends are exactly as fragile as any connection pool.

Health checks: the contract that separates percentiles from outages

Load balancing guarantees nothing about a backend being healthy — it can only remove backends it has evidence about. The evidence is the health check:

• Liveness: TCP connect succeeds, or HTTP 2xx returned. Tells you "the process accepts connections", which is weaker than it sounds — a process can accept connections while its database pool is exhausted and every request will hang.
• Readiness: a more application-aware probe — "can you do real work right now?" This is the signal that matters for traffic, and it's the one most teams don't implement. Kubernetes calls this readinessProbe for exactly that reason.
• Draining: the balancer must drain (stop sending new work, let in-flight finish) before removing a backend — which is preStop in Kubernetes, and a graceful-stop handshake in every serious balancer.

Read that last one again: the checking logic is only half the contract. The other half is deprovisioning, and it's where the classic incident lives: backend removed from the rotation while a migration is mid-flight, clients tail-domain N socket waits, percentiles go to the moon, and the diagram's magic box takes the blame.

Stickiness is a promise you should almost never keep

"Send the same client to the same backend" — session stickiness — is the safety valve for stateful servers and the cause of the worst failure mode in the catalog: hot-spot overload. One user's workload (a heavy tenant, a pathological session) pins a backend while its siblings idle. And stickiness defeats the balancer's reasons to exist: the scale answer — add capacity — stops helping, because new backends receive no sticky traffic.

The engineering answer is to design so that stickiness is unnecessary: shared session store, replicated cache, idempotent writes. If you cannot avoid stickiness (real-time WebSocket state, in-process gaming session data), then bound it: consistent hashing with a bounded hash-ring window gives you "most of the time the same backend, occasionally a different one" — stable under churn, better than absolute stickiness.

The failure modes that survive all optimizations

1. The healthy-looking dead backend — accepts connections, hangs forever. Health checks must measure work, not connectivity.
2. Cascading drain — one backend dies, its traffic redistributes, and the redistribution overloads the survivors; then a survivor dies, and the cascade repeats. This is the pattern behind most "the whole fleet fell over in thirty seconds" postmortems.
3. The hash collision of fate — consistent hashing concentrates a specific client group on one backend after fleet changes. Ring smoothing (virtual nodes) is the fix.
4. Symmetric, not redundant — two balancers in active/passive "HA" are a failed component the moment one is down. Active/active with real routing work is what redundancy means.