The Runtime Theory
ApplicationDSAnetwork

Webhook Delivery Trace: Outbox, Sender, Retry Schedule, and Dead Letter

A step-by-step walk from event commit to webhook HTTP delivery, retry scheduling, idempotency headers, and the dead-letter queue.

The Runtime Theory Team3 min read07 steps

trace spine

  1. 01 Business event commits with outbox row
  2. 02 Relay publishes outbox rows to a queue
  3. 03 Sender worker formats the webhook
  4. 04 HTTP POST is delivered with signature
  5. 05 Failure classified and scheduled for retry
  6. 06 Retry backoff escalates
  7. 07 Max attempts reached: dead letter
On this page

A webhook is a promise made without a handshake: "when this happens, I will call your URL." This trace follows one event from the database transaction that created it to the moment a dead-letter queue admits that a consumer's endpoint is never going to accept it.

1. The business event commits with an outbox row

order.paid happens. The payment handler does two writes in one transaction: the order status update and an outbox row (event_id, type='order.paid', payload, created_at). This is the entire trick of the transactional outbox: the event cannot exist without the business fact that produced it, and the business fact cannot commit without its event. No dual-write race, no lost events between a committed order and a delivery system that never heard about it.

2. The relay publishes outbox rows

A relay worker polls the outbox table (every 100ms-1s, or tails the replication stream): SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 500. Each row is published to the delivery queue and marked published_at = now(). If the relay dies mid-publish, the rows stay unpublished and are re-picked — at-least-once semantics begin here. The outbox itself is the durability anchor: everything after it is best effort over HTTP, which is all HTTP can be.

3. The sender worker formats the webhook

The queue hands the message to a sender worker. The worker resolves the consumer's registered endpoint and secret, and builds the POST: the event payload, an X-Webhook-Signature (HMAC-SHA256 of the body with the shared secret — the consumer's only way to verify the call came from you), and critically an Idempotency-Key: <event_id>. The consumer dedupes on that key; without it, your retries (which are coming) are indistinguishable from duplicate events.

4. The HTTP POST goes out

The worker sends POST https://consumer.example.com/hooks/order.paid with a generous timeout (10-30s). Three outcomes: 2xx — done, acknowledge, delete from queue. Non-2xx — the consumer exists but rejects the event. Network failure — timeout, connection reset, DNS failure: the request is ambiguous; it may or may not have arrived (this is where the consumer's idempotency handling earns its keep). Every non-success is a retry candidate.

5. Failure is classified and scheduled

The worker never retries synchronously in a tight loop — it computes the next attempt and schedules it. Common schedule (Stripe's is roughly this shape, in minutes):

text
attempt 1: +1 min      attempt 4: +10 min
attempt 2: +5 min      attempt 5: +30 min   (cap)
attempt 3: +10 min     attempt 6: +60 min

The message goes back to the queue with a visibility delay (SQS) or to a retry queue. Each scheduled attempt is separate from the original — the queue holds the retry state, so a worker crash doesn't lose the schedule. Retries are throttled, not stormed: if your consumer is down, you deliver 1 attempt/minute, not 60.

6. The backoff escalates

Each attempt repeats the same trace: dispatch, HTTP, classify. The signature and idempotency key are constant across attempts — every retry is a replay of the same logical event, which is the point. The worker also tracks why it fails: a 410 Gone (endpoint deleted) is different from a 500 — the first can be dead-lettered immediately.

7. Max attempts reached: the dead letter

After the budget (6 attempts, or ~2 hours, configurable), the message moves to the dead-letter queue (DLQ). Delivery stops — no more hammering an endpoint that has failed six times. The DLQ is not the end: it is the handoff to a human or a repair process. A monitor alerts on DLQ depth; an operator replays the batch after the consumer fixes their endpoint; a tool lets you inspect the payload to decide whether the consumer even deserves a replay.

The cost summary

Per event: one transactional outbox write (a few ms, amortized), one relay read, one queue round trip, one HTTP POST (~50-200ms at p50, 10-30s on failure). The design's entire value is in the boundaries: the outbox makes events durable before delivery exists, the retry schedule makes failure slow and polite, and the DLQ makes persistent failure visible and replayable.