Queueing Theory for Engineers

Queueing theory is the closest thing systems engineering has to thermodynamics: it is a set of laws that hold no matter what your stack is, and it explains the shape of every latency graph you have ever seen — including why a service at 90% utilization takes ten times longer per request than an idle one.

Little's Law: the conservation equation

Little's Law is the conservation of requests: the number of requests in a system equals the arrival rate multiplied by the time each one spends inside.

L = λ × W
in-flight = QPS × mean latency

It holds for any stable system, with no assumptions about arrival patterns or service times. Its practical use is instantaneous sizing arithmetic: if a service averages 200ms latency at 1,000 QPS, there are 200 requests in flight. If that service runs 50 worker threads, the other 150 requests are sitting in a queue — and their latency is the queue's fault. This is the fastest capacity audit you can run: multiply QPS by latency, compare against your concurrency limit (thread pool, DB pool, gRPC max streams), and you know exactly where the queueing starts.

Utilization: the dimensionless dial

Utilization ρ is arrival rate divided by service rate — the fraction of time a worker is busy:

ρ = λ / μ     (μ = 1 / mean service time)

A service that takes 5ms per request has μ = 200 req/s. Feed it 100 req/s and ρ = 0.5; 180 req/s and ρ = 0.9. Utilization is the master variable because response time depends on it — catastrophically.

M/M/1: the 10x at 90%

The simplest model — Poisson arrivals, exponential service times, one server, unbounded queue — has a closed-form response time:

T = 1 / (μ − λ) = T_service / (1 − ρ)

The queueing delay is ρ / (1 − ρ) × T_service. That fraction is the entire story:

At 90% utilization, one request in ten arrives to find the worker busy, and — this is the feedback loop — the queue that forms slows mean response time, which by Little's Law raises in-flight requests, which lengthens the queue further. The M/M/1 formula is the closed form of that feedback: latency blows up as 1/(1−ρ), and the curve is unbounded as ρ → 1.

The assumptions, and why the lesson survives them

M/M/1 assumes Poisson arrivals (real traffic is bursty — flash crowds, retry storms, thundering herds) and exponential service times (real service times have variance, sometimes enormous). Both assumptions make reality worse, not better: burstier arrivals and higher service variance shift the curve left, so the knee lands below the M/M/1 prediction — around 70-80% utilization for well-behaved services, lower for noisy ones. Multiple workers (M/M/c) stretch the curve right but don't remove it: a 10-worker pool at ρ = 0.9 still pays the 10x penalty, just at a bigger absolute load.

The engineering takeaways are qualitative and robust:

• The knee exists. Utilization is not linear in latency. Operating a latency-sensitive service above ~70-80% utilization is a deliberate latency purchase.
• Utilization explains tail latency. The 99th percentile is dominated by queueing, so the p99 multiplier is larger than the mean multiplier — a 10x mean penalty at 90% can be 50x at p99.
• Autoscaling should target utilization, not QPS. Adding nodes keeps ρ in the flat zone. A "scale at 65% CPU" rule is queueing theory with a threshold.

A tiny computation to keep around:

def mm1(service_ms, arrival_per_s):
    mu = 1000.0 / service_ms            # requests per second
    rho = arrival_per_s / mu
    if rho >= 1: return None            # unstable: queue grows forever
    t_ms = service_ms / (1 - rho)
    q = rho * rho / (1 - rho)           # mean requests waiting in queue
    return rho, t_ms, q
 
for a in (100, 140, 160, 180, 190):
    print(a, mm1(5, a))

The outputs are the whole course: 11.1ms, 16.7ms, 25ms, 50ms, 100ms total latency on a 5ms service — at 95% utilization, 19x the idle time in queueing alone.

Where the model breaks, and what to do

Two real-world corrections. First, unbounded queues are fiction: real systems drop or reject, which is a relief — rejection converts infinite queueing delay into a bounded, honest failure (see backpressure and flow control). Second, response time is not the only resource: Little's Law applies per resource. A 5ms API calling a 40ms database has two queues in series, and the utilization that matters is the database's — queueing behind a saturated dependency propagates up the stack.

Queueing theory's practical gift is the vocabulary: utilization is a dial, latency is the consequence, and the knee of the curve is where capacity planning and autoscaling should live.