Latency, Throughput, and Queueing Are Linked

Latency measures how long one operation takes; throughput measures how many operations complete per unit time. They feel independent and are not: past a certain utilization, extra concurrency stops adding useful output and starts adding waiting. Nearly every "we added load and everything got slow" incident is this link being discovered the hard way.

Service time vs response time

Separate two quantities that dashboards routinely conflate:

• Service time — time actually working on a request (CPU, I/O once scheduled).
• Response time — service time plus queueing time before you get a worker.

A system can have high throughput and terrible tail latency because requests spend their life in a queue. Consider a single worker whose service time is 10 ms — capacity is 1 / 0.010 = 100 req/s. What arrival rate does to the response time (M/M/1 intuition, where wait grows as ρ/(1−ρ) × service time):

arrival λ   utilization ρ   queue wait ρ/(1-ρ)×S   response ≈ wait + S
  50 req/s     0.50            10 ms                 20 ms
  70 req/s     0.70            23 ms                 33 ms
  90 req/s     0.90            90 ms                100 ms
  95 req/s     0.95           190 ms                200 ms
 100 req/s     1.00           unbounded — backlog grows without end

Throughput rises with load right up to capacity and then flatlines; latency rises the whole time and then explodes. Operating at 95% utilization is a decision to have 20 ms requests become 200 ms the instant traffic wobbles.

Backlog arithmetic: when service can't keep up

If a service processes 500 req/s but receives 700, the queue grows by 200 jobs every second until arrivals fall, requests are shed, or the service fails. The three knobs are the only ones that exist: add capacity, reduce arrivals, or reject work.

Little's Law ties the three measurable quantities together for a stable system:

L = λ × W
 
L = average number of items in the system
λ = arrival rate
W = average time a item spends in the system
 
example: λ = 250 jobs/s, W = 4 s average in queue  →  L = 1,000 jobs resident
if you observe 1,000 jobs and 250/s but latency
reads 40 ms, one of your three numbers is wrong —
Little's Law is the consistency check on your dashboards.

The check question falls straight out of it:

2,000 jobs queued, drain at 250/s, arrivals paused:
  drain time = 2,000 / 250 = 8 seconds
 
if arrivals continue at 200/s:
  net drain = 250 - 200 = 50/s → 2,000 / 50 = 40 seconds
  and if arrivals ever reach 250/s, the backlog never drains at all:
  the queue is stable only while λ < capacity.

Concurrency beyond capacity only lengthens the queue

The instinct under load is to raise parallelism — more workers, bigger pools, more threads. Through the capacity ceiling, that adds in-flight requests, not completions:

fixed capacity 100 req/s, service 10 ms
clients allowed in flight:      throughput    p99 response
        10                       95 req/s      ~15 ms
       100                       99 req/s      ~60 ms      (queuing starts)
       500                      100 req/s     ~490 ms      (fully saturated)
     5,000                      100 req/s     ~4.9 s       (now it's a 5 s timeout)

This is also why client timeouts make throughput worse under overload: requests that would have completed at second 4 are abandoned at second 1, the work is still done, and the retry multiplies arrivals — a retry storm on top of a saturated queue.

Averages hide the burst

Average latency stays flat while users suffer, because bursts and dependency stalls land in the tail. If 99% of requests take 20 ms and 1% take 400 ms, the average is ~24 ms; double the tail to 800 ms and the average moves to ~28 ms (+17%) while p99 doubles. Capacity planning from averages systematically undersizes the system for exactly the minutes that matter — and the queue is the tail: a 2-second burst of 3× arrivals into a service running at 70% utilization creates a backlog whose drain time outlives the burst.

The corrective is admission control: a bounded queue with a deliberate rejection policy.

bounded queue (cap 500) + shed policy:
  depth < 400            → accept
  depth 400–500          → accept only cheap/idempotent classes, reject the rest
  depth = 500            → 503 + Retry-After, fail fast in <1 ms
 
  result: admitted requests keep ~33 ms response times,
  rejected ones get an immediate, honest error — instead of
  everything (including health checks) waiting 5 seconds

An unbounded queue converts overload into memory pressure and then into an OOM; a bounded queue converts overload into a signal your clients can act on. Bufferbloat — in networks, in thread pools, in Kafka consumer lag — is the same lesson in three costumes.

Key takeaways

• Latency = service time + queue time. Past ~70–80% utilization, queueing dominates and tail latency degrades long before throughput does.
• Little's Law (L = λW) is a consistency check on your own dashboards: three numbers, one identity, and a mismatch means a measurement bug.
• Beyond capacity, concurrency only lengthens the queue. Adding workers is a capacity decision; adding in-flight requests is not.
• Bounded queues plus load shedding preserve the served majority. Fail fast and let clients retry elsewhere instead of letting everything wait.
• Plan for p95/p99 and bursts, not averages — the average is the one number the users don't experience.

Google SRE Book: Monitoring Distributed Systems covers tail latency and utilization consequences in depth.