A webhook is a server-to-server callback: your system owes a subscriber a POST. The interesting machinery is what happens when that POST fails — and it will fail. Trace one event through the delivery system.
trace stepSystem
An event fires (payment succeeded, repo pushed). The webhook worker writes the delivery record —
{id: evt_123, url, payload, attempts: 0, next_attempt_at: now} — to a persistent delivery queue. Nothing is sent at event time unless the queue is empty; enqueue is the contract, send is the work.trace stepSystem
The worker pops the record and sends
POST https://subscriber.example/hook with the payload, a timestamp, and a signature header (X-Hub-Signature-256: sha256=HMAC(secret, timestamp + body)). The signature is the subscriber's only way to prove the request is genuine — computed from the shared secret, over both the timestamp (anti-replay) and the body (anti-tamper).trace stepSystem
The subscriber is down, or returns 500, or — worse — the network times out (2s, 5s, 10s default). Any non-2xx is a delivery failure. 4xx responses are special: a 400 (bad payload) or 404 (dead URL) is not retryable — retrying a permanent rejection is a waste — and the event should go straight to inspection, not the ladder.
trace stepSystem
The worker computes the next attempt from the schedule: Stripe's shape is 1min → 5min → 30min → 2h → 5h → 12h → 24h+. The backoff is exponential with a long tail, and the
Retry-After header — if the subscriber is alive enough to send one — overrides the schedule. Each rung is a fresh POST with the same event id.trace stepSystem
When the subscriber is finally up, it verifies the HMAC (constant-time compare), checks the timestamp window (reject anything older than ~5 minutes), and dedupes: the event id in the payload is the idempotency key, so attempt 4 arriving after attempt 2 was already processed returns a replayed response instead of re-processing the payment.
trace stepSystem
The delivery succeeded, or the schedule ran out. The ladder has a terminal rung: after N attempts (Stripe: ~10 over 3 days; GitHub: 3) the event is marked failed and moves to the dead-letter queue. The DLQ is not the end — it's the start of a human or automated review loop: check the subscriber's endpoint, fix the payload, replay.
trace stepSystem
The DLQ record keeps every response received across all attempts (
last_attempt_status, last_error, attempt timestamps). This history is the diagnostic gift: a 500 with a stack trace on attempt 3 tells you the subscriber was up and the bug is theirs. From here, events are replayed, expunged, or routed to an alert.enqueue → attempt (POST + signature) → 2xx → done
→ 4xx → dead-letter (permanent rejection)
→ 5xx/timeout → next rung of ladder
ladder exhausted → dead-letter with full attempt historyThe economics: one event, up to ~10 attempts over three days, each attempt costing one POST plus one queue record. The mechanism's value is in the middle of the ladder — the window where a subscriber that was down for an hour comes back and still receives every event it missed, in order, with proof of authenticity. The DLQ is the system's way of being honest: some deliveries are permanently lost, and the queue is where the apology is written down.