Health checks are the load balancer's evidence-gathering mechanism — the only way it knows a node is dead. And yet, in every fleet, there is a period where a dead node keeps receiving traffic. Understanding health checks means understanding the timing of that blindness: how long detection takes, how long removal takes, and why both windows exist by design.
Active vs passive: probing vs observing
Active checks are probes: the balancer (or orchestrator) sends a synthetic request — an HTTP GET /healthz — on an interval (typically 5-10s; Kubernetes default periodSeconds 10) and marks the node unhealthy after N consecutive failures (Kubernetes failureThreshold 3). Detection time ≈ interval × threshold (30 seconds at defaults), plus a startup grace period. Active checks catch hard deaths quickly and work with no real traffic — a necessary feature for a low-traffic service at 2am.
Passive checks observe real traffic: the balancer or client measures failures in the actual request stream — 5xx ratio, connection timeouts, response-time degradation. The signal is richer — the actual workload, not synthetic — and detection is faster for real failures. The cost: you need traffic to detect failure. A dead node serving 1 QPS takes minutes to accumulate a threshold (a 1% error budget at 10 requests/minute is hours).
Production systems run both: active checks drive routing decisions (who gets new traffic), passive observations drive client-side decisions (circuit breakers) and feed alerting.
Why a down node still gets traffic
The four mechanisms, in rough order of how often they cause incidents:
1. The detection window. With interval 5s and threshold 2, a node that dies at t=0 keeps receiving traffic until t≈10s — longer if the balancer waits for in-flight drains. At 1,000 QPS that's 10,000 requests into a dead connection pool, all timing out and retrying. The window is inherent: every active-check policy states "how much traffic am I willing to send to a corpse?"
2. Liveness ≠ readiness. The probe checks that the process is alive (accepts connections); the process's ability to do work is a different question — one it can fail independently (DB pool exhausted, disk full, dependency down). A /healthz that returns 200 without touching its dependencies reports "the process has a pulse," which the load balancer happily interprets as "route 100% of traffic here." The fix is the readiness-probe contract: check the dependencies the request path actually uses, with a timeout small enough to not become the new bottleneck (a DB ping with a 50ms budget, a cache ping, a disk-write check on stateful nodes).
3. The check ≠ the real request. The probe path is synthetic and often misses the real failure: a health endpoint that doesn't authenticate, doesn't read the database, doesn't touch the queue. The classic incident: /healthz returns 200 for two days while every real request 500s on an auth dependency the probe never exercises.
4. Stale routing state everywhere else. The balancer removing a node only affects traffic that passes through it. Clients with cached DNS (TTL 30-300s), client-side load balancers (gRPC resolver refresh, service-mesh caches), and long-lived connections (keep-alive, WebSocket) keep pointing at the dead node after the balancer ejects it. Hence drain semantics: stop new connections and wait out in-flight ones — and client libraries need their own failure detection, not just the fleet-level signal.
Circuit breakers: the passive complement
Circuit breakers are the client-side half of health checks: instead of a balancer probing, each client counts failures on real requests (e.g., 5 failures in a 10-second window) and "opens" — stops sending traffic to that dependency for a cooldown (e.g., 30s), then "half-opens" to probe with a single request before closing again. Their crucial property: they react to degradation, not just death. A node at 95% error rate trips the breaker while every active health check still passes. The interplay: the balancer's probes decide fleet-level routing; the client's breakers decide per-call routing — and a fleet that relies on only one of the two has a detection blind spot.
The health check as telemetry
The cheapest observability win is treating the probe as a measurement. The check response should carry structured state: dependencies checked with their own latencies, plus node saturation (queue depth, pool utilization) — turning the endpoint into a per-node snapshot the orchestrator already polls. Kubernetes livenessProbe / readinessProbe / startupProbe formalize this: startup gates traffic until warm, liveness restarts the unrecoverable, readiness removes the temporarily-unsafe from routing.
readinessProbe:
httpGet:
path: /healthz?full=1
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # node is routed out ~10s after losing its dependenciesWhat "healthy" actually means
A healthy node is one whose real request path works — not one that answers a synthetic probe. Make the check faithful: exercise the dependencies the request path uses, keep it cheap enough to run every few seconds, and treat "removed from rotation" as a routing event with its own latency. The down node still gets traffic because detection, removal, and propagation are three separate timers — the system only becomes resilient when all three are designed.