Retries are the difference between a flaky dependency and an outage. But a retry loop that isn't disciplined becomes a retry storm — everyone retrying in lockstep can take down the very service you're trying to reach. The trace:
trace stepSystem
The client sends request 1; it gets a 500, a timeout (client-side deadline of, say, 2s), or a connection refused. The error class decides what happens next: 5xx and network errors are retryable; 4xx are not (retrying a 400 will 400 again); a 429 or 503 with a
Retry-After header must be respected, not improvised.trace stepSystem
The client computes
delay = min(cap, base × 2^attempt). With base 100ms, attempts land at 100ms, 200ms, 400ms, 800ms, 1600ms, capped at 5s (or the AWS-style 20s). This is where naive implementations die: without a cap, attempt 10 waits 51 seconds; with a cap, the spacing plateaus.trace stepSystem
The delay becomes
random(0, delay) (full jitter) or delay/2 + random(0, delay/2) (equal jitter). The math matters: without jitter, 1,000 clients that failed at the same moment all retry at the same moment, so the second wave is as synchronized as the first. Full jitter spreads the wave across the full backoff window — the retry rate decays smoothly instead of spiking.trace stepSystem
The client re-issues the request with the same idempotency key (or the same request body for idempotent verbs — GET, PUT, DELETE). The server sees a duplicate. This is the moment the receiver's dedupe machinery earns its keep: the second arrival of an idempotent request must not double-charge or double-apply.
trace stepSystem
Every loop iteration checks the budget:
deadline = start + 30s (duration budget) or attempts < 5 (count budget). The budget exists so a user request dies instead of hanging: a 30s budget at the above schedule allows ~5–6 attempts total. A request that would exceed the budget is not retried — it returns the last error.trace stepSystem
The client surfaces the failure (with the last attempt's status) and the caller's own timeout machinery takes over. Downstream, the request may fan out into the caller's retry — which is why distributed budgets (e.g., gRPC's
RetryBudget limiting retries to 10% of request volume) exist: each layer's retry must not multiply the layers beneath it.attempt, delay = 0, base
while attempts < max and now < deadline:
resp = send(request) # ~1 RTT + server time
if ok: return resp
if resp.headers["Retry-After"]: delay = that value
else: delay = random(0, min(cap, base * 2**attempt))
sleep(delay)
attempt += 1
return last_errorThe numbers that matter: base 100ms, cap 5–20s, full jitter, a duration budget of 30s (5–6 attempts), and a hard rule that only idempotent requests are retried. That combination keeps a failing dependency costing you ~30s of latency, not a stampede and not a hang — and it is exactly what every SDK from AWS to gRPC ships by default.