The Runtime Theory
mediumApplicationDSA#backend#webhooks#messaging#reliability

Explain webhooks and why delivery is hard

Tests whether you understand webhooks as an at-least-once delivery problem with no acknowledgement channel — retries, signatures, idempotent receivers, and replay.

The Runtime Theory Team2 min readasked at stripe · twilio · shopify

This question tests whether you understand webhooks as a push with no pull — the receiver can't ask again, so the sender carries all the delivery responsibility. The interviewer wants the delivery lifecycle walked end to end, with every step that can fail named.

The mental model: at-least-once delivery over a channel with no acknowledgement.

The flow: an event occurs (charge succeeded, invoice paid), the provider writes it to a queue, a dispatcher POSTs a signed payload to your registered URL. Delivery is hard for three structural reasons.

First, there is no synchronous ack. HTTP returns a status code, but the receiver can be down, the request can time out after the receiver already processed it, or a proxy can swallow the response. The sender cannot distinguish "not delivered" from "delivered, response lost." That forces at-least-once semantics: the provider retries with exponential backoff across a delivery window — Stripe retries for roughly three hours at increasing intervals — and every receiver must be idempotent, keying on the event ID in the payload.

Second, ordering is not guaranteed across retries. Events can arrive out of order and duplicated, so the receiver's storage must key on event ID, and the processing logic must be commutative or versioned — never "last one wins" on a raw timestamp.

Third, the receiver is an untrusted, moving target. Signature verification — an HMAC over the raw body with a shared secret — is how the receiver knows the payload is authentic; without it, a webhook URL is a free callback-injection vector. And receivers are slow paths: acknowledge quickly and process asynchronously, because a slow receiver backs up the provider's dispatcher, and providers will eventually mark the endpoint failing, route events to a dead-letter queue, and email you to fix it.

Tradeoffs and edge cases: providers offer manual replay from a dashboard — that's re-delivery of old events, so handlers must be re-runnable and ignore stale payloads via a timestamp field. The hardest case is partial failure: the event was processed but the response was lost, so the receiver gets a duplicate. Receiver-side idempotency on event ID is the entire game — everything else is sender-side best effort.

This answer walks

Follow-ups they'll push on

  1. 01Why can't a provider deliver webhooks exactly once?
  2. 02What does the receiver do with an event it has already processed?
  3. 03How would you verify a webhook payload is authentic?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.