The Runtime Theory
mediumApplicationDSA#backend#resilience#retries#reliability

Design a retry strategy that doesn't make an outage worse

Probes your understanding of failure propagation — retryable vs. permanent errors, backoff with jitter, circuit breakers, and the load multiplier math behind retry storms.

The Runtime Theory Team2 min readasked at stripe · netflix · google

This question is really testing whether you understand failure propagation: retries are the first thing that turns a small outage into a full meltdown. The interviewer wants two rules — retry only retryable failures, and add jitter — plus the mechanism that stops retries from amplifying load.

The mental model: a retry is a decision with a budget.

First, classify the failure. A timeout or a 502 from the load balancer is retryable. A 400 with a validation error is not — retrying reproduces the same failure and wastes work. A 409 or 429 carries meaning (already processed, rate-limited), so retrying without reading the response is how you double-charge a customer.

If it's retryable, the schedule matters. Fixed-interval retries synchronize: every client retries on the same tick, and a recovering server receives a wall of requests at once — the thundering herd. Exponential backoff spreads them: delay = base × 2^attempt, capped, plus jitter. Jitter is not polish; without it, retries from a fleet of clients cluster on the same powers of two. A realistic schedule: 100ms, then a few hundred ms with jitter, doubling to a 5–10s cap, with a hard maximum of 3–5 attempts total.

The load multiplier is the math the interviewer wants: 10,000 clients each retrying 4 times multiplies traffic 5×. That's why the second mechanism exists — the circuit breaker. After N consecutive failures, or a failure-rate threshold over a sliding window, the client stops sending entirely and returns an error immediately for a cooldown period; in half-open state it probes with a trickle of traffic to decide whether to close or trip again. The breaker converts load amplification into load shedding.

Tradeoffs and edge cases: retries only make sense for idempotent operations — pair every retry design with an idempotency key so the duplicate lands exactly once. Spread retries across hosts and availability zones so a zone-wide failure doesn't retry into the same dead zone. Per-operation retry budgets (e.g., 1s total) bound worst-case latency for interactive requests, while background jobs get the long-horizon backoff. And the closing point: when the server is already failing, your retry policy should reduce offered load, never increase it.

This answer walks

Follow-ups they'll push on

  1. 01Which HTTP status codes are safe to retry, and which aren't?
  2. 02Why is jitter not optional when a fleet of clients retries?
  3. 03Where does the circuit breaker live — client, server, or both?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.