How does a load balancer know a backend is healthy?

This question tests whether you understand that "health" is a measurement of serviceability, not liveness — and that every check mechanism has a blind spot. A strong answer covers active checks, passive detection, and the semantics of failure.

The mental model: three layers of "healthy," each cheaper and less accurate.

1. Process liveness — the cheapest signal: is the port open? A TCP connect probe (SYN → SYN-ACK) says the kernel is accepting connections. It says nothing about the app: a handler can be deadlocked while the socket accepts fine. TCP checks catch "server crashed" and miss "app hung."

2. HTTP-level health checks — the load balancer sends a real request to a designated path (e.g. /healthz) every N seconds (typically 5-10s, with a timeout of ~2-3x the expected latency). The backend answers 200 with a body it controls — the app itself can gate that endpoint on its dependencies: DB connectivity, queue depth, config load. This is where the check becomes honest: the app declares itself healthy only if it can actually serve traffic.

3. Passive detection — no probe at all. The load balancer watches the data path: connection failures, TCP resets, RST on established connections, repeated 5xx responses, response-time thresholds. If a backend fails a window of these (e.g. 3 consecutive errors in 30s), it's ejected. This catches what probes miss: a backend that passes /healthz but fails on real requests — because the health endpoint is itself cached or bypasses the broken code path.

The decision mechanics: the load balancer marks a backend unhealthy only after failure_threshold consecutive failures, and removes it from rotation only for new connections — established connections keep draining (that's the "drain"/connection-draining window, typically 30-60s, so in-flight requests complete). Recovery mirrors it: success_threshold consecutive successful probes before traffic resumes, to avoid flapping a backend in and out every interval.

Tradeoffs and edge cases: check interval and timeout set a tradeoff — a 5s interval means a dead backend still receives ~5s of requests. Too-aggressive checks cause cascades: if the health endpoint shares a dependency with real traffic (e.g. it queries the DB and the DB is slow), every backend fails at once and the load balancer drops the whole pool. Healthy-but-slow backends are usually caught by response-time-based ejection rather than probes. And on the return path, the load balancer's own health check traffic can keep TCP connections alive and hide keep-alive exhaustion — which is why you see health checks reused as keep-alive keepers.