Logs tell you what one machine did. Metrics tell you what the fleet did on average. Neither can answer the question that every outage starts with: what did THIS request experience? That is the job of distributed tracing — stitching logs from every hop a single request touched into one timeline, so "the checkout is slow" becomes "the 4% of requests with a 2-second auth delay are all timing out against the identity service."
The three ingredients
A trace is built from spans. A span is a named unit of work with a start and end time, an error status, and attributes. One trace = one logical request = a tree of spans, glued together by two ids:
- trace_id: the same random 128-bit id on every span in the request. This is the join key.
- span_id: unique per span; each span records its parent_span_id, which rebuilds the tree. The first span (no parent) is the root span.
The protocol that does this is OpenTelemetry's W3C Trace Context header. The entry point of a request puts two values into traceparent — trace-id and span-id — and every downstream service reads that header, creates its own child span, and passes the header on. Done correctly, a chain of five services produces one trace with five (or more) spans.
Instrumentation: you do not want to write it by hand
Manual startSpan/endSpan calls everywhere is a maintenance trap. The OpenTelemetry approach is auto-instrumentation: the HTTP client and server libraries do the propagation for you (injecting and reading traceparent), and automatic spans capture the URL, status, and timings. What you add by hand is the semantic detail — the SQL query, the cache key, the queue name — within the named span:
ctx, span := tracer.Start(ctx, "checkout.charge")
span.SetAttributes(attribute.String("payment.gateway", "stripe"), attribute.Int("amount.cents", 1999))
defer span.End()The span attributes are what turn a generic "HTTP request" span into "the Stripe charge for order 42".
Sampling: the cost you must decide
Capturing every span costs real money: a hot API at 10,000 rps, with an average of 4 spans per request, generates 40,000 spans/second — 2.4 million per minute — and storage and egress are not free. The standard answer is probabilistic head sampling: keep, say, a 5% uniform sample of trace IDs. Because the decision is made at the trace root and propagated in the context, whole traces are kept or dropped together, so the data you do have is complete.
Two production patterns that beat pure random sampling:
- Tail-based sampling (decide after the spans arrive): keep everything for a short window, then keep traces that ended in error, exceeded latency thresholds, or carried rare attributes — errors always sampled, happy paths sampled at 1%. Guarantees you can debug your failures, not only your average.
- Be careful with naive "sample interesting errors only". Rare fast failures are exactly what you want, and a metrics-level counter alone will not show you the request that hit three different pods.
From traces to production answers
A trace is most valuable when it fills the gap between metrics and the request reality:
- The outlier, not the p99. The metric says p99 is 180 ms; the trace shows the user clicking again at 4 seconds. Correlate the slowest spans by service to see the dependency that jitters — often a downstream timeout at 4.999 s hiding behind a 5 s deadline.
- The leaky fan-out. A request that fans out to 20 shards looks fine per-shard and slow end-to-end; the trace shows the slowest parallel child dominating the parent span.
- Connection-pool stalls. The span says "waiting in queue" for 900 ms before the DB call — a symptom the metrics dashboard never showed you because the database itself was fast.
The questions that mean observability is working
- Can I, for any user-reported symptom, find the trace_id? (Toolbar capture on the frontend: yes.)
- Can the trace show me every queue wait, network hop, and retry, not only the spans I remembered to add?
- Is every production incident drill-downable to a single span with its attributes, not a pile of correlated log lines?
Key takeaways
- Trace id + span id + parent span id rebuild the full request journey across services; propagate them in
traceparent. - Prefer auto-instrumentation for propagation, and add spans only for semantics worth filtering on.
- Sample whole traces head-of-trace for volume, tail-of-trace for correctness — always keep error traces.
- Judge traces by whether they answer the three drill-down questions above.
A trace is finished being well-engineered when a 3 a.m. page becomes "which span, which attribute, which threshold" instead of "which team should wake up".
Production cost model and latency numbers
To apply Distributed Tracing and Observability: Follow the Request Across Systems effectively in production systems, engineers must move beyond qualitative diagrams and quantify the exact hardware and network costs at each boundary:
| Operation / Boundary | Typical Latency | Dominant Bottleneck | Production Mitigation |
|---|---|---|---|
| L1 / L2 Cache Access | 1 ns – 4 ns | Register dependency chains & pointer chasing | Contiguous memory layouts, cache-line alignment (64 B), and batch iteration |
| Main Memory (DRAM) Fetch | 60 ns – 100 ns | TLB misses, random heap indirection, NUMA hops | HugePages (2 MB), arena/slab allocators, and pre-allocated ring buffers |
| Kernel Syscall / Context Switch | 1 µs – 5 µs | User-to-kernel mode switch, page-table flush, lock contention | Non-blocking readiness loops (epoll / io_uring), batching, and zero-copy I/O |
| NVMe SSD Random Read / fsync | 50 µs – 500 µs | Block I/O queue depth, write amplification, WAL flushes | Group commit, sequential append logs, and bounded buffer-pool pinning |
| Intra-Region RPC Roundtrip | 0.5 ms – 2.5 ms | Serialization, TCP/TLS framing, tail queueing ($L = \lambda W$) | Connection pooling, deadline propagation, and p95 hedged requests |
Failure modes, debugging, and operational checklist
In real incidents, systems rarely fail at average (p50) load—they fail when a slow dependency, skewed hot key, or resource leak pushes concurrency across a non-linear saturation cliff. When reviewing code or architecture that relies on Distributed Tracing and Observability: Follow the Request Across Systems, walk through three verification questions:
- What is the invariant under partial failure or contention? If a worker crashes mid-operation, a network packet is delayed and reordered, or two threads race on the same state transition, prove which atomic primitive, version number, or idempotency key prevents silent corruption.
- How is backpressure enforced when demand exceeds capacity? Unbounded in-memory queues, unthrottled retries, and blocking calls without explicit deadlines turn a localized
100 msslowdown into a fleet-wide Out-Of-Memory (OOM) cascade. Always bound queue depths, propagate caller deadlines, and shed excess load early. - Which metrics and traces isolate the bottleneck in under 60 seconds? Instrument saturation, queue wait time vs service execution time, cache hit/miss ratios, and error rates by failure reason so on-call engineers can immediately distinguish a compute bottleneck from an I/O or lock-contention stall.