A webhook is a promise made without a handshake: "when this happens, I will call your URL." This trace follows one event from the database transaction that created it to the moment a dead-letter queue admits that a consumer's endpoint is never going to accept it.
1. The business event commits with an outbox row
order.paid happens. The payment handler does two writes in one transaction: the order status update and an outbox row (event_id, type='order.paid', payload, created_at). This is the entire trick of the transactional outbox: the event cannot exist without the business fact that produced it, and the business fact cannot commit without its event. No dual-write race, no lost events between a committed order and a delivery system that never heard about it.
2. The relay publishes outbox rows
A relay worker polls the outbox table (every 100ms-1s, or tails the replication stream): SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 500. Each row is published to the delivery queue and marked published_at = now(). If the relay dies mid-publish, the rows stay unpublished and are re-picked — at-least-once semantics begin here. The outbox itself is the durability anchor: everything after it is best effort over HTTP, which is all HTTP can be.
3. The sender worker formats the webhook
The queue hands the message to a sender worker. The worker resolves the consumer's registered endpoint and secret, and builds the POST: the event payload, an X-Webhook-Signature (HMAC-SHA256 of the body with the shared secret — the consumer's only way to verify the call came from you), and critically an Idempotency-Key: <event_id>. The consumer dedupes on that key; without it, your retries (which are coming) are indistinguishable from duplicate events.
4. The HTTP POST goes out
The worker sends POST https://consumer.example.com/hooks/order.paid with a generous timeout (10-30s). Three outcomes: 2xx — done, acknowledge, delete from queue. Non-2xx — the consumer exists but rejects the event. Network failure — timeout, connection reset, DNS failure: the request is ambiguous; it may or may not have arrived (this is where the consumer's idempotency handling earns its keep). Every non-success is a retry candidate.
5. Failure is classified and scheduled
The worker never retries synchronously in a tight loop — it computes the next attempt and schedules it. Common schedule (Stripe's is roughly this shape, in minutes):
attempt 1: +1 min attempt 4: +10 min
attempt 2: +5 min attempt 5: +30 min (cap)
attempt 3: +10 min attempt 6: +60 minThe message goes back to the queue with a visibility delay (SQS) or to a retry queue. Each scheduled attempt is separate from the original — the queue holds the retry state, so a worker crash doesn't lose the schedule. Retries are throttled, not stormed: if your consumer is down, you deliver 1 attempt/minute, not 60.
6. The backoff escalates
Each attempt repeats the same trace: dispatch, HTTP, classify. The signature and idempotency key are constant across attempts — every retry is a replay of the same logical event, which is the point. The worker also tracks why it fails: a 410 Gone (endpoint deleted) is different from a 500 — the first can be dead-lettered immediately.
7. Max attempts reached: the dead letter
After the budget (6 attempts, or ~2 hours, configurable), the message moves to the dead-letter queue (DLQ). Delivery stops — no more hammering an endpoint that has failed six times. The DLQ is not the end: it is the handoff to a human or a repair process. A monitor alerts on DLQ depth; an operator replays the batch after the consumer fixes their endpoint; a tool lets you inspect the payload to decide whether the consumer even deserves a replay.
The cost summary
Per event: one transactional outbox write (a few ms, amortized), one relay read, one queue round trip, one HTTP POST (~50-200ms at p50, 10-30s on failure). The design's entire value is in the boundaries: the outbox makes events durable before delivery exists, the retry schedule makes failure slow and polite, and the DLQ makes persistent failure visible and replayable.