Circuit Breakers and Timeouts: Protecting Services from Cascading Failure

When a downstream service degrades, the failure does not stay there. Requests pile up, their timeouts lengthen, threads and connections on your service exhaust, your health checks start failing, and the load balancer starts failing you — the cascade. The three mechanisms that stop it are timeouts, retries with budgets, and circuit breakers. Each has a distinct job: timeouts bound how long a call may run, retries convert transient failures into success, and the circuit breaker stops sending work to a peer that has already proven it can't handle it.

Timeouts: the primary bound

The first mechanism is the timeout — a hard upper bound on call duration. Its job is to convert "we'll wait forever" into "we'll wait N milliseconds." Every outbound call needs three: a connect timeout, a request timeout, and an overall deadline that survives retries.

// per-call timeouts: connect, request, and an overall deadline
client := &http.Client{
    Timeout: 5 * time.Second,           // whole request: dial + write + read
    Transport: &http.Transport{
        DialContext: (&net.Dialer{
            Timeout: 2 * time.Second,   // connect
        }).DialContext,
    },
}
 
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
defer cancel()
req := http.NewRequestWithContext(ctx, "POST", svcURL, body)

Two mechanisms matter beyond the timeout itself. Deadlines must be propagated: the same context/deadline flows into the downstream call, so your downstream doesn't keep working after you've given up (and so the overall call is bounded by the outermost deadline, not the sum of inner ones). And timeout values must be chosen from real percentiles: a p99 of 400ms means a 1s timeout is generous; a 5s timeout on that path means 99% of calls are waiting 10× longer than necessary, tying up threads for no reason.

Retries: the budget

A retry is correct only when the failure is transient (connection reset, 503, 429) and the operation is safe to repeat. The two mechanisms that keep retries from turning into amplification are budgets and exponential backoff with jitter.

The amplification math: if every client retries 3 times, a downstream outage multiplies inbound load by 4× at the worst moment — this is how a single slow service takes down the whole fleet. Google's SRE guidance ("Addressing cascading failures") recommends retry budgets: retry only when the local retry rate is a small fraction (e.g. 10%) of recent request volume — typically enforced by the client library tracking failures vs. successes over a rolling window.

def should_retry():
    # budget: allow retries only if recent attempts failed at a low rate
    recent = attempts_window.sample()
    return recent.failures / max(recent.total, 1) < 0.10

Backoff: sleep between attempts growing geometrically (1s, 2s, 4s) plus jitter so synchronized clients don't retry in lockstep — the "thundering herd" of retries. Every retry needs its own time budget drawn from the same overall deadline, or retries multiply the wall-clock cost.

Circuit breaker: the three states

A circuit breaker is a state machine in the caller that stops sending traffic to a failing peer. It has exactly three states:

• Closed — normal operation. Every call counts; failures increment a rolling window counter (typically the failure rate over the last N seconds, not a raw count).
• Open — the failure threshold was crossed (e.g. >50% of requests failing over 30 seconds). Calls fail fast before hitting the network — no timeout wait, no wasted threads — returning an immediate error or fallback.
• Half-open — after a cooldown period (the retry interval, e.g. 30–60s), the breaker lets a small probe of traffic through (a few requests, or one at a time) to test recovery. If the probes succeed, it closes; if any fails, it opens again.

// hysteresis: open only after the failure *rate* clears the threshold,
// and only after enough calls exist to be statistically meaningful
if stats.total >= minRequestCount && stats.failureRate > 0.5 {
    state = Open
    openedAt = time.Now()
}
if state == Open && time.Since(openedAt) > cooldown {
    state = HalfOpen
}

The two threshold parameters — the failure threshold and the minimum request count — work together. A raw count opens the breaker on two failures in a quiet service; a pure rate opens it on two failures out of two requests. Standard libraries (Hystrix, resilience4j, Polly, github.com/sony/gobreaker) all expose both: the rule of thumb is a failure rate that is high enough to mean "this peer is actually broken" (50% is common) and a minimum volume high enough that the rate is meaningful (10–20 requests per window).

Where the mechanisms compose

The three mechanisms are a pipeline, not alternatives. Timeout bounds the individual call; retry budget bounds the amplification of retries; the breaker bounds sustained exposure to a broken peer; and once the breaker is open, calls fail fast — which means the caller's timeout never even fires, because there's no call.

One design point that separates production systems from toys: the fallback. An open circuit should return something — a stale cached value, a default, a queued job, a clear 503 — not propagate the error as if the breaker were the outage. The breaker contains the outage; the fallback keeps it from becoming user-visible.

Tuning in production

• Start with conservative numbers: 1s request timeouts, 3 retries with backoff, 50% failure rate over a 30s window, 30s cooldown.
• Verify with chaos: kill one replica of a dependency and watch the breaker open and recover; tune the cooldown until recovery doesn't flap.
• Instrument every state transition — breaker open/close/half-open events with timestamps are the single best signal for when the dependency actually died.