Health checks are the load balancer's evidence-gathering mechanism — the only way it knows a node is dead. And yet, in every fleet, there is a period where a dead node keeps receiving traffic. Understanding health checks means understanding the timing of that blindness: how long detection takes, how long removal takes, and why both windows exist by design.
Active vs passive: probing vs observing
Active checks are probes: the balancer (or orchestrator) sends a synthetic request — an HTTP GET /healthz — on an interval (typically 5-10s; Kubernetes default periodSeconds 10) and marks the node unhealthy after N consecutive failures (Kubernetes failureThreshold 3). Detection time ≈ interval × threshold (30 seconds at defaults), plus a startup grace period. Active checks catch hard deaths quickly and work with no real traffic — a necessary feature for a low-traffic service at 2am.
Passive checks observe real traffic: the balancer or client measures failures in the actual request stream — 5xx ratio, connection timeouts, response-time degradation. The signal is richer — the actual workload, not synthetic — and detection is faster for real failures. The cost: you need traffic to detect failure. A dead node serving 1 QPS takes minutes to accumulate a threshold (a 1% error budget at 10 requests/minute is hours).
Production systems run both: active checks drive routing decisions (who gets new traffic), passive observations drive client-side decisions (circuit breakers) and feed alerting.
Why a down node still gets traffic
The four mechanisms, in rough order of how often they cause incidents:
1. The detection window. With interval 5s and threshold 2, a node that dies at t=0 keeps receiving traffic until t≈10s — longer if the balancer waits for in-flight drains. At 1,000 QPS that's 10,000 requests into a dead connection pool, all timing out and retrying. The window is inherent: every active-check policy states "how much traffic am I willing to send to a corpse?"
2. Liveness ≠ readiness. The probe checks that the process is alive (accepts connections); the process's ability to do work is a different question — one it can fail independently (DB pool exhausted, disk full, dependency down). A /healthz that returns 200 without touching its dependencies reports "the process has a pulse," which the load balancer happily interprets as "route 100% of traffic here." The fix is the readiness-probe contract: check the dependencies the request path actually uses, with a timeout small enough to not become the new bottleneck (a DB ping with a 50ms budget, a cache ping, a disk-write check on stateful nodes).
3. The check ≠ the real request. The probe path is synthetic and often misses the real failure: a health endpoint that doesn't authenticate, doesn't read the database, doesn't touch the queue. The classic incident: /healthz returns 200 for two days while every real request 500s on an auth dependency the probe never exercises.
4. Stale routing state everywhere else. The balancer removing a node only affects traffic that passes through it. Clients with cached DNS (TTL 30-300s), client-side load balancers (gRPC resolver refresh, service-mesh caches), and long-lived connections (keep-alive, WebSocket) keep pointing at the dead node after the balancer ejects it. Hence drain semantics: stop new connections and wait out in-flight ones — and client libraries need their own failure detection, not only the fleet-level signal.
Circuit breakers: the passive complement
Circuit breakers are the client-side half of health checks: instead of a balancer probing, each client counts failures on real requests (e.g., 5 failures in a 10-second window) and "opens" — stops sending traffic to that dependency for a cooldown (e.g., 30s), then "half-opens" to probe with a single request before closing again. Their crucial property: they react to degradation, not only death. A node at 95% error rate trips the breaker while every active health check still passes. The interplay: the balancer's probes decide fleet-level routing; the client's breakers decide per-call routing — and a fleet that relies on only one of the two has a detection blind spot.
The health check as telemetry
The cheapest observability win is treating the probe as a measurement. The check response should carry structured state: dependencies checked with their own latencies, plus node saturation (queue depth, pool utilization) — turning the endpoint into a per-node snapshot the orchestrator already polls. Kubernetes livenessProbe / readinessProbe / startupProbe formalize this: startup gates traffic until warm, liveness restarts the unrecoverable, readiness removes the temporarily-unsafe from routing.
readinessProbe:
httpGet:
path: /healthz?full=1
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # node is routed out ~10s after losing its dependenciesWhat "healthy" actually means
A healthy node is one whose real request path works — not one that answers a synthetic probe. Make the check faithful: exercise the dependencies the request path uses, keep it cheap enough to run every few seconds, and treat "removed from rotation" as a routing event with its own latency. The down node still gets traffic because detection, removal, and propagation are three separate timers — the system only becomes resilient when all three are designed.
Production cost model and latency numbers
To apply Health Checks and Observability effectively in production systems, engineers must move beyond qualitative diagrams and quantify the exact hardware and network costs at each boundary:
| Operation / Boundary | Typical Latency | Dominant Bottleneck | Production Mitigation |
|---|---|---|---|
| L1 / L2 Cache Access | 1 ns – 4 ns | Register dependency chains & pointer chasing | Contiguous memory layouts, cache-line alignment (64 B), and batch iteration |
| Main Memory (DRAM) Fetch | 60 ns – 100 ns | TLB misses, random heap indirection, NUMA hops | HugePages (2 MB), arena/slab allocators, and pre-allocated ring buffers |
| Kernel Syscall / Context Switch | 1 µs – 5 µs | User-to-kernel mode switch, page-table flush, lock contention | Non-blocking readiness loops (epoll / io_uring), batching, and zero-copy I/O |
| NVMe SSD Random Read / fsync | 50 µs – 500 µs | Block I/O queue depth, write amplification, WAL flushes | Group commit, sequential append logs, and bounded buffer-pool pinning |
| Intra-Region RPC Roundtrip | 0.5 ms – 2.5 ms | Serialization, TCP/TLS framing, tail queueing ($L = \lambda W$) | Connection pooling, deadline propagation, and p95 hedged requests |
Failure modes, debugging, and operational checklist
In real incidents, systems rarely fail at average (p50) load—they fail when a slow dependency, skewed hot key, or resource leak pushes concurrency across a non-linear saturation cliff. When reviewing code or architecture that relies on Health Checks and Observability, walk through three verification questions:
- What is the invariant under partial failure or contention? If a worker crashes mid-operation, a network packet is delayed and reordered, or two threads race on the same state transition, prove which atomic primitive, version number, or idempotency key prevents silent corruption.
- How is backpressure enforced when demand exceeds capacity? Unbounded in-memory queues, unthrottled retries, and blocking calls without explicit deadlines turn a localized
100 msslowdown into a fleet-wide Out-Of-Memory (OOM) cascade. Always bound queue depths, propagate caller deadlines, and shed excess load early. - Which metrics and traces isolate the bottleneck in under 60 seconds? Instrument saturation, queue wait time vs service execution time, cache hit/miss ratios, and error rates by failure reason so on-call engineers can immediately distinguish a compute bottleneck from an I/O or lock-contention stall.