A request is not one thing. It is a sequence of discrete hops, each with its own cost, and the client-visible latency is the sum. This trace follows one GET /users/42 from a browser to a database and back, and accounts for every millisecond.
1. Client initiates request
The client (browser, mobile app, or another service) builds the HTTP request line, headers, and body. At this moment nothing has left the machine. The only cost so far is in-memory serialization: sub-millisecond. If the client is a browser, there is usually no existing connection yet, so it must do a full connection setup.
2. DNS resolution and connection setup
The client resolves api.example.com. If the answer is not in the local DNS cache, a recursive resolver round trip costs 5-30ms. Then TCP: a three-way handshake, one full round trip to the server, ~20-60ms across a continent. Then TLS 1.3: one additional round trip. Total connection setup: ~2 RTTs, typically 40-120ms, paid once and then amortized by keep-alive. The server may be behind multiple layers — in front of the load balancer, not the origin.
3. Load balancer terminates
The load balancer accepts the TCP connection and terminates TLS (that's why it needs the certificate). It applies a routing decision — path-based, header-based, or weighted — and forwards to a backend. An L7 LB (NGINX, Envoy, ALB) parses the full HTTP request to route; an L4 LB (AWS NLB, IPVS) forwards packets without reading them. L7 cost: 0.1-0.5ms per request in software; the added latency of the proxy hop itself is usually under 1ms, but the second connection leg (LB → backend) may reuse an existing backend connection via keep-alive.
4. Kernel accepts into the socket backlog
The backend's kernel completes the handshake (or accepts on an existing connection), pushes the request into the socket receive buffer, and wakes the event loop. Under load, the accept queue — somaxconn / net.core.somaxconn, typically 128-4096 — fills; overflow SYN packets are dropped, not queued, which is why backends under sudden load show connection timeouts rather than slow responses. Queuing delay here grows as requests / (capacity - requests); at 90% utilization the wait doubles.
5. Handler dispatches to the middleware chain
The framework (Express, Fastify, Django, Spring) routes the request to a handler wrapped in middleware. Middleware runs in registration order: CORS, request ID assignment, body parsing, auth, rate limiting. Each middleware is a function call chain — typically 0.2-1ms total for cheap ones. The request ID generated here is what lets you trace the request through logs later.
6. Middleware does its work
- Auth middleware: verifies the JWT — signature check (HS256 HMAC or RSA verify, ~10-50µs) and expiry check (integer comparison). No network call.
- Rate limiter: if token-bucket backed by Redis, one round trip to the cache: 0.3-1ms.
- Body parsing: JSON.parse of a 1KB body, ~10-20µs.
- If auth hits an upstream session service instead of verifying locally, add a full network round trip: 1-10ms. This is why "middleware calls a service" is the classic latency-budget killer.
7. Business logic queries the database
The handler builds a SQL query and takes a connection from the pool (see the connection pool trace for what happens when none are free). The query executes: planner decides the access path (~50-200µs), the executor reads pages from the buffer pool — 0.1-1ms if cached, 1-10ms if a disk read is required. The row is fetched, serialized into a tuple, and returned over the wire. A typical point lookup: 1-5ms total. A poorly indexed scan: 50-500ms.
8. Response serialized and returned
The framework serializes the result to JSON (a 1KB object, ~50-100µs), possibly compresses it with gzip or brotli (~0.5-2ms CPU), and writes it out. The reverse path repeats: backend → LB → client, mostly on the already-established connections. The client receives the response and renders.
The budget
A healthy p99 request at ~100ms total typically decomposes roughly as: 40ms connection setup (amortized away by keep-alive), 1-5ms LB, 1-3ms middleware, 1-10ms DB, 1-5ms serialization/compression, and the rest — 50-80ms — is network round-trip time. When you see a slow request, the first question is always: which hop owns the time?