The requirement is exactly-once effect with at-least-once delivery. The interviewer wants the mechanism, not the promise: a unique constraint, a status machine, and a stored response.
Requirements and capacity
5M charges/day (~58/s average, 10x peak ≈ 600/s). Clients retry aggressively on timeout — mobile networks and gateway timeouts make that the common case, not the edge case. Contract: no double charge, and every retry returns the original result, not a new attempt.
The mechanism
Client sends Idempotency-Key: <uuid> on POST /charges. The server stores UNIQUE(merchant_id, idempotency_key) — the database is the arbiter. The app never "decides" two requests are the same; the index does, atomically, under concurrency.
Write path
INSERT charges(id, merchant_id, amount, currency, status='pending', idempotency_key, created_at). If the key exists (unique violation, or SELECT-first then INSERT), return the stored response — a replay gets the original 200/402/500, and the response blob was stored with the row.- The pending row is a lock on intent:
UPDATE charges SET status='authorized' WHERE id=? AND status='pending'— the conditional guard makes concurrent retries serialize: exactly one transition wins, every other request sees the current state and either waits or returns it. - Call the provider, store
provider_refand the response blob. A timeout here is not a failure — it is an unknown, and the design must never guess. - Provider webhook → reconcile worker flips
captured/settled. The reconcile path is what makes timeouts safe: the charge is settled by the provider's signal, not by the client's retry.
Data model
charges table plus an append-only events ledger (every status transition — the audit and reconciliation source of truth). Two writes, one transaction: the charge row and its events must commit together. The dual-write problem (DB + provider call) is solved with the outbox pattern: persist the intent row first, a worker performs the provider call — so a crash between the two never leaves a committed charge with an un-replayable provider call.
Retention
Idempotency keys expire (24h to 7d). After expiry, the same key is a new charge by definition — state the policy and the boundary risk explicitly: the window must be longer than the slowest legitimate retry loop.
Failure modes
- Leader failover: the uniqueness must be enforced by the DB (unique index), not an app-level lock in one process — with replication, uniqueness is enforced at the primary and the index carries through replicas (database-replication-topologies).
- Provider charged, DB lost the response: reconcile worker compares provider state against
authorizedrows past their timeout — the webhook plus periodic statement reconciliation is the safety net. - Retry storm on one key: hot unique index — bounded by the conditional-update guard; the row is touched once, replays are read-only.
Bottlenecks
The unique index under retry storms; provider latency holding the row in pending; reconcile scans; retention cleanup. None of these are hard — the hard part was the status machine, and it is already done.