Every network is lossy. When your client times out waiting for a response, it cannot know whether the server applied the request and lost the response, or never received the request at all. So it retries. And now you have a choice: the payment was charged once and the client believes it failed, or the payment was charged twice and the client believes one succeeded. Retry safety — exactly-once effect, not exactly-once delivery — is the mechanism that makes the first outcome impossible and the second the default.
The industry-standard mechanism is the Idempotency-Key header: a client-generated opaque string sent with the request. The server stores the key, the request, and the response; when a duplicate key arrives, it replays the stored response without re-executing the operation.
Why "at-least-once" forces the design
TCP gives you reliable, ordered delivery, but application-level timeouts, load balancer retries, and client backoff all reintroduce duplicates. Stripe sends Idempotency-Key requests to roughly 0.5% of their API calls as duplicates — a small percentage that would otherwise double-charge customers thousands of times a day.
The mechanism works because a duplicate request is indistinguishable from a new one by content alone. Retrying a POST /payments with identical payload is not safe — a new authorization can succeed twice. The only way to distinguish a retry from a new intent is a client-supplied token with agreed-upon uniqueness.
What the server must store
A correct implementation keeps a small table with three things:
CREATE TABLE idempotency_records (
key_hash text PRIMARY KEY, -- key + operation idempotent pair
request jsonb NOT NULL, -- full request body
response jsonb, -- stored response
status int, -- HTTP status of stored response
created_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz NOT NULL
);Key storage rules that matter:
- Key the pair, not the key alone.
user_id + idempotency_key— a key must never collide across customers, and should be scoped to one operation type. - Hash the key. Never store raw keys; they often embed PII like email addresses.
- Store the request body. If the client retries with the same key but a different body, that is a protocol violation — 422 with a clear error, never a silent replay of the first response.
- Purge after expiry. Records are useless after the replay window and become a security liability. TTL or a cron sweep both work.
Replay windows
Keys must expire: storing them forever turns the table into a growing append-only log. Stripe's default window is 24 hours; AWS recommends 5–10 minutes for most Idempotency-Key use. The window bounds how long the client may retry — set client retry budgets inside the server window, or a late retry executes the operation a second time.
A common refinement is per-request windows: payment intents get 24h, search analytics get 1 minute. The window lives in the record, not the schema.
Concurrency: the lost update
The failure mode that destroys naive implementations is the concurrent duplicate. Two retries of the same request arrive within milliseconds of each other. Both threads read the table, neither finds a record, both execute the operation, both write. You now have two payments.
The fix is atomic key acquisition — one insert, winner takes all:
def acquire_idempotency(key_hash, request):
try:
db.execute(
"INSERT INTO idempotency_records (key_hash, request) VALUES (?, ?)",
(key_hash, request),
)
return "execute", None # we won; run the operation
except IntegrityError:
row = db.execute(
"SELECT response, status FROM idempotency_records WHERE key_hash = ?",
(key_hash,),
).fetchone()
return "replay", row # someone else ran it; replayThe insert either wins or loses — there is no read-then-write race. For distributed databases with weaker isolation, a FOR UPDATE lock on the key row, or a dedicated idempotency table in a strongly consistent store (DynamoDB's conditional put_item, Redis SET NX), gives the same guarantee.
Idempotent operations that need no keys
Some operations are safe to retry with zero infrastructure: GET, PUT (full replacement), DELETE on an already-deleted resource, and any operation whose effect is derived purely from its input. The idempotency key exists precisely for the operations that aren't — charges, increments, enqueues, sends, reservations.
Verify it with curl
curl -X POST https://api.example.com/v1/charges \
-H "Idempotency-Key: ck-7f3a1e" \
-d '{"amount": 5000, "currency": "usd"}'
# retry the identical request; the server must return the
# same response body and status, without a second charge
curl -X POST https://api.example.com/v1/charges \
-H "Idempotency-Key: ck-7f3a1e" \
-d '{"amount": 5000, "currency": "usd"}'If the second response differs in status code, response body, or a server-side side effect occurred twice, the implementation is broken.
What retry safety does not give you
Exactly-once effect scoped to one operation is not the same as cross-operation guarantees. A key does not make two different operations atomic: the charge succeeds, the notification fails, and no key reconciles them — that needs an outbox or a saga. And a key replays the stored response; if your cached response format changes mid-window, replays are still stale. Keyed retries are the foundation, but exactly-once at the system level is a composition of idempotency, atomic writes, and transaction logs — never a single header.
The checklist
- Client generates a new key per operation, never reuses keys across operations.
- Key is scoped per user + operation type, stored hashed, purged after the window.
- Duplicate acquisition is atomic — INSERT with conflict handling, not check-then-write.
- Same key + different body → 422; same key + same body → replay stored response.
- Client retry budget is strictly shorter than the server replay window.