Distributed Calls Can Fail Halfway Through

In a distributed system, a caller may lose contact with a process that is still running, or receive no response after the remote side completed its work. A network partition, an overloaded queue, a crashed process, and a slow dependency all look identical from the caller's side: no answer in time. The design consequence is that timeout is not failure — it is uncertainty, and uncertainty needs its own handling.

Four outcomes, one observation

caller sends request, waits, times out:
 
  1. request never arrived; nothing happened           → safe to retry
  2. request arrived, failed before any effect         → safe to retry
  3. request arrived, effect happened, response lost   → retry = duplicate
  4. request arrived, still being processed            → retry = duplicate
                                                         (and added load)
 
observation: identical in all four cases (deadline exceeded)

Cases 3 and 4 are the ones that break systems, and no timeout tuning can distinguish them — that is the two-generals problem wearing an incident number. The caller's honest response is to report uncertainty, not to invent an answer.

Deadline propagation: one budget, not per hop

A client starts a request with a deadline. Without propagation, each hop applies its own timeout and the total wall-clock bound multiplies:

service A (timeout 1 s) → B (timeout 1 s) → C (timeout 1 s) → D (timeout 1 s)
 
without propagation: worst case 4 s of caller waiting, and A's client
  gave up at 1 s — so A keeps working on a request nobody wants,
  holding a connection pool slot for 3 more seconds
 
with propagation: A attaches remaining budget; each hop reads it and
  uses min(own timeout, remaining)
  A→B gets 1000 ms, B→C gets 750 ms, C→D gets 500 ms
  total bounded by the original 1 s, and every downstream
  deadline expires *before* the caller's — so nobody wastes work

Concretely: grpc-timeout and HTTP/2's own deadline mechanisms carry the budget in the protocol; otherwise pass an explicit header (X-Deadline-Ms) and subtract as you go. A useful invariant: the remaining budget at each hop must be less than the caller's remaining budget, or the tail of the system will do work that has already been abandoned.

Cancellation is the other half, and it is best-effort by nature: when the deadline fires, the client cancels local waiting; whether the server notices depends on transport propagation and on the server checking for cancellation between units of work.

Too short is a failure mode too

no timeout at all     → threads, connections, and pool slots held indefinitely;
                        one slow dependency exhausts the whole service
timeout too short     → false failures under normal variance, plus duplicate
                        retries that add load exactly when the dependency is slow
timeout right-sized   → derived from your p99 budget and measured service times,
                        e.g. p99 200 ms end to end → 120 ms for a remote dependency

False timeouts are not free errors: each one is a retry that the server may still be serving. The correct sizing is empirical — set from the dependency's observed p99 plus margin, review after load tests, and never accept a framework default (30 s, 60 s) that was chosen for a different system. Note the asymmetry: a timeout too long wastes resources on work nobody wants, while one too short manufactures duplicates and load — both feed the same failure spiral, in opposite directions.

Timeouts also compose badly across layers unless someone owns the budget explicitly. A client with a 1-second deadline calling a service that allows 900 ms internally, which calls a dependency with a 800 ms budget, leaves the last hop 100 ms to do real work — and the moment the dependency needs 150 ms, every request fails even though no single layer was misconfigured in isolation. Budgets must shrink monotonically down the call chain.

Correlate the attempt, dedupe the operation

Logs and traces need a request identifier to stitch attempts together across services: a trace ID travels with the request and ties each hop's spans into one path. But a trace ID identifies an execution path, not a logical action — it changes per retry. Idempotency identifies the action across retries, which is what case 3 actually requires.

The check question makes the distinction concrete: a worker times out while charging a card, then retries.

attempt 1: POST /charges  idempotency key = "ord_739"   → server charges card,
                                                          response lost in transit
attempt 2: POST /charges  idempotency key = "ord_739"   → server looks up key,
  sees a stored result for ord_739, returns the SAME charge receipt
  without touching the payment provider
 
resolution comes from the record, not from the network:
  INSERT charges (idem_key, request_hash, status, response)
  PRIMARY KEY (idem_key)   -- attempt 2 hits the conflict → replay stored response

Without that record, attempt 2 charges twice and no amount of tracing will tell you which of the two charges was "the" one until the customer emails you.

Key takeaways

• Timeout means "unknown," not "failed." Design the caller to handle four outcomes, and retry only when the operation has an identity that makes duplicates detectable.
• Propagate one deadline through every hop. Per-hop timeouts multiply total latency and keep downstream work running after the caller has left.
• Right-size timeouts from measurements — too long exhausts resources, too short manufactures duplicates; both are worse than the failure you were defending against.
• Trace IDs correlate executions; idempotency keys identify actions. You need both, and they are not interchangeable.

Google SRE Book: Addressing Cascading Failures covers timeout budgets and load shedding under partial failure.