Retries Need an Operation Contract

A timeout tells a client it did not receive a response in time. It does not tell the client whether the server performed the operation. The server may have completed a payment, written a row, or started a job immediately before the response was lost. Safe retries therefore cannot be a client-side policy bolted on later — they require an operation contract that the server participates in.

Retry safety follows semantics

Repeating a read is often safe, though its result may have changed. Repeating a write creates duplicate effects unless the API defines what a retry means. HTTP gives partial guidance (GET/PUT/DELETE are defined as idempotent, POST is not), but the real question is about side effects: does running this operation twice charge twice, send twice, increment twice?

An idempotency key makes the logical action identifiable across retries:

$ curl -X POST https://api.example.com/charges \
    -H "Idempotency-Key: ord_739_attempt1" \
    -H "Content-Type: application/json" \
    -d '{"order_id": 739, "amount_minor": 4999, "currency": "USD"}'
 
# first call : charges the card, stores (key → response), 201
# retry (same key, same body) : returns the STORED response, no new charge
# reuse (same key, different body) : 409 conflict — the key identifies
#                                    one request, not one endpoint

Server-side, the key and the effect must be coordinated in one transaction, or a crash between them reintroduces the very duplication the key was meant to prevent:

BEGIN;
INSERT INTO idempotency_keys (key, request_hash, status, response, created_at)
VALUES ('ord_739_attempt1', 'sha256:9f2c…', 'in_progress', NULL, now());
-- on unique violation: read the stored row → if done, replay response;
--                       if in_progress, either wait or return 409/retry-later
INSERT INTO charges (order_id, amount_minor, provider_ref) VALUES (739, 4999, 'ch_8821');
UPDATE idempotency_keys
   SET status = 'done', response = '{"id":"ch_8821","status":"succeeded"}'
 WHERE key = 'ord_739_attempt1';
COMMIT;

Scope the key to an account (so one caller cannot replay another's operation), store enough request identity to reject accidental reuse with different parameters, and retain the record for at least your client's maximum retry horizon (24 hours is a common floor).

Timeouts and backoff are load controls

Set a timeout for the whole operation and a separate, smaller one for each dependency call. Then retry only what may recover, with a budget:

retryable                : timeouts, connection resets, 429, 502/503/504
not retryable            : 400, 401, 403, 409, 422 — the request is the problem
honor                    : Retry-After on 429/503
 
full jitter (AWS recipe): sleep = random(0, min(cap, base × 2^attempt))
 
attempt   window (base=100 ms, cap=10 s)
   1      0–100 ms
   2      0–200 ms
   3      0–400 ms
   4      0–800 ms

Jitter is not a nicety: without it, 10,000 clients that failed together retry together — the synchronized retry storm, which arrives as a square wave exactly on the dependency's recovery boundary and knocks it back down.

Amplification is multiplicative across layers, which is why the cap matters more than the backoff:

one user action
  → API retries 3×        (each) 
  → each fans out to 3 downstream calls, retried 3× each
  worst case: 3 × 3 × 3 = 27 requests per user action
at 500 user actions/s → 13,500 req/s against a dependency sized for 500

Retry only at one layer where you can see the whole operation — or make every layer's budget explicit and tiny (1 retry, no inner retries).

Backpressure: refuse early instead of queueing late

Circuit breakers, bulkheads, and queue limits prevent one unhealthy dependency from consuming the whole service. From the caller's side, the same policy is visible as admission control: when your own queue depth or in-flight count crosses a threshold, return 429/503 fast (under a millisecond) instead of accepting work that will wait five seconds and then fail anyway. Fast rejection lets well-behaved clients back off and reroute; slow acceptance converts their retries into a thundering herd against an already-degraded path.

Keep the boundary visible

Backend reliability also depends on input validation, authorization before any side effect, transaction boundaries, and logs that name the operation. And keep two identifiers straight — they answer different questions:

trace ID       : "which execution path did this attempt take?"   changes per retry
idempotency key: "which logical action is this?"                 stable across retries

A trace ID in a log helps you reconstruct what happened; only an idempotency key in the data store lets the system decide what happened. Confusing the two produces the classic post-mortem line: "we could see all three attempts in the trace, but nothing told us the second one had already charged."

Key takeaways

• A retry contract is server state, not client hope. Keys scoped to the caller, request identity checked, outcome stored, key + effect committed atomically.
• Classify failures before retrying them: transport/5xx/429 yes, 4xx no; honor Retry-After; one retry layer per operation.
• Full jitter or synchronized storms — with base 100 ms and cap 10 s, jitter spreads the herd instead of doubling the outage.
• Budget retries multiplicatively: three layers at three attempts is 27× the load; cap it at one layer or make inner budgets nearly free.
• Refuse early under load. 429 in a millisecond is a gift; a 5-second queue that ends in 503 is a retry amplifier.

The AWS Builders' Library guide to idempotent APIs describes the ambiguity caused by a lost response and the server-side contract needed for safe retries.