Retries are the difference between "one failed call" and "a failed call that failed four more times, taking down the caller's caller." This trace follows a failed HTTP request through the retry machinery: classification, backoff, budget, and idempotency — the four things that make retries safe instead of suicidal.
1. The request fails
A call to a downstream service (payments, search, a third-party API) returns an error. Not all errors are equal. A 503 from a healthy-but-overloaded server is transient; a 400 "invalid request" will fail identically every time you replay it; a connection reset is ambiguous — the request may have reached the server and the response was lost. The first decision is classification: only retry what is plausibly transient.
2. Error classification
Typical retryable: 429, 503, 504, timeouts, connection resets. Typical fatal: 400, 401, 403, 422, malformed responses. A 409 is special: it often means a previous attempt of the same request already succeeded — the strongest signal that idempotency (step 4) matters. Retrying a fatal error is not harmless; every attempt consumes downstream resources and every retry of a 4xx pollutes logs and error budgets.
3. Backoff is computed
Naive retry: try again immediately. That fails at scale, because every concurrent client retries at the same moment — a thundering herd. The standard schedule is exponential with a cap:
attempt 0 (first retry): base × 2^0 = 100ms
attempt 1: 200ms
attempt 2: 400ms
attempt 3: 800ms
attempt 4: 1600ms
attempt 5: 3200ms (cap at 30s from here)With full jitter, instead of sleeping the full amount, the client sleeps a uniform random value between 0 and the scheduled delay. This is the single most important line in retry code: without jitter, 100 clients retrying 10 failed requests each fire 1000 simultaneous attempts at identical offsets. With full jitter, the same 1000 attempts spread across the whole window. AWS's AWS SDK and Google's gRPC retry policies both use jitter for exactly this reason.
4. The retry fires, protected by an idempotency key
If the operation is not read-only, each retried attempt must be indistinguishable to the downstream from a duplicate — or the downstream must make it so. The client generates an idempotency key (a UUID per logical operation) and sends it on every attempt:
POST /api/payments
Idempotency-Key: 9f1c2b6e-7d3a-4f8e-9b0c-2a5d6e7f8a1b
{"amount": 4900, "currency": "USD"}The server stores the key and the response for the first processed attempt (TTL 24h, in a cache or DB). A retry with the same key is answered with the stored response — 200 for the payment attempt that actually succeeded, even if the original response was lost in transit. Without a key, a timeout followed by a retry can charge a card twice.
5. The budget is checked
Every retry loop needs a ceiling. Two budgets combine: time (a deadline — "give up after 10 seconds", regardless of attempts) and count (max attempts, typically 3-5). The budget must also account for the rest of the request's lifecycle: a retry loop that takes 8 seconds of a 10-second API deadline leaves 2 seconds for the response path. A common failure mode is nested retries — the service retries for 10s, then its caller retries the whole call for 10s, and the client times out at 5s with three layers still grinding.
6. Budget exhausted: surface the error
When the budget runs out, the error must propagate with context, not be silently swallowed: attempt count, total time spent, the last error. The retry did its job — it absorbed transient failure. Persistent failure is now someone else's problem, and if the system has a circuit breaker, this is the signal that trips it: after enough consecutive failures, the breaker opens and requests fail fast without even attempting the call.
The one sentence summary
Retry only transient failures, wait exponentially with jitter, carry an idempotency key, spend from a fixed budget, and when the budget is gone — fail loudly so a human or a breaker decides what happens next.