Circuit Breaker: Failure Count, OPEN State, Half-Open Probe, Recovery
A step-by-step walk from the first 500s through the trip threshold, the fast-fail window, and the half-open probe that decides recovery.
The Runtime Theory Team··3 min read·07 steps
layer stack
System
HWHardware
KKernel
RTRuntime
APPApplication
SYSSystem
CLIClient
NETNetwork
TLSCrypto
SRVServer
adjacent altitudes in this subsystem are still being traced
trace spine
01 failures start accumulating
02 threshold is crossed
03 circuit goes OPEN
04 requests fast-fail
05 settle timer runs
06 half-open probe
07 recovery or re-trip
A circuit breaker is a client-side tripwire: when a dependency starts failing, stop calling it before you make it worse. The mechanism is a three-state machine — CLOSED, OPEN, HALF-OPEN — and this is the walk through one complete failure episode.
trace stepSystem
The dependency (say, a payment API) starts returning 500s and timeouts. The circuit breaker in the client is CLOSED: every call passes through, and every outcome is recorded in a sliding window — Resilience4j counts the last N calls (e.g. 100); Hystrix counts the last 10s. At 20% failures the circuit is still CLOSED; it is measuring, not acting.
trace stepSystem
Failures reach the trip threshold — e.g. 51 of the last 100 calls failed. The breaker transitions to OPEN. The trip is a decision, and it's a local one: this client's view of the dependency's health. Other clients may still be CLOSED, which is fine — circuit state is per-client evidence, not global truth.
trace stepSystem
Now the interesting part: the breaker stops trying. Calls are rejected immediately without touching the network — fail-fast with a CircuitOpenException in ~0ms instead of a 2s timeout. The dependency's load from this client drops to zero. This is the mechanism's whole point: a dead dependency is not asked for retries; the client's timeout cost is replaced by an instant refusal.
trace stepSystem
While OPEN, every call fails fast. The caller sees an error immediately and can run its fallback — a cached response, a degraded path, a "try later" reply. In Hystrix terms this is the fallback execution, and it's what makes the user experience "slow feature" instead of "hung page." The OPEN state also stops the retry storm before it starts.
trace stepSystem
The circuit stays OPEN for a configured window — typically 5–30s (Hystrix: 5s; Resilience4j: configurable waitDurationInOpenState). During this window the breaker performs no health checks; it's waiting out whatever the failure is. The window is the one place the breaker trades availability for certainty.
trace stepSystem
The timer expires; the breaker moves to HALF-OPEN and allows a single probe request (or a small limited batch — Resilience4j allows maxCallCount in half-open). This probe is a deliberate health check disguised as a real request: exactly one attempt, full timeout, real dependency. The rest of traffic is still fast-failing.
trace stepSystem
Probe succeeds → the circuit returns to CLOSED: the window resets and traffic flows again. Probe fails (timeout, 500) → the circuit snaps back to OPEN for another settle window — often longer, because the dependency has now failed for a while (some implementations double the window per re-trip). Either way, at most one request reached the sick dependency during the whole OPEN period.
text
CLOSED: count failures in sliding window → rate > 50% → OPENOPEN: reject all calls (~0ms) for waitDuration → HALF-OPENHALF-OPEN: allow 1 probe → success → CLOSED | failure → OPEN
The cost ledger: while CLOSED, one counter update per call (~microseconds). While OPEN, calls cost ~0ms and the dependency sees zero load. The half-open probe is the only real request during the outage. That is the entire economics of the circuit breaker: spend a few microseconds of bookkeeping to convert a multi-second timeout storm into an instant, bounded, self-healing failure.