Capacity Planning: The Math

Capacity planning sounds like guessing, but it's arithmetic with three inputs you can actually measure: your traffic (QPS), your latency (the service time per request), and your utilization target (how close to the cliff you're willing to live). The math is Little's Law and queueing theory wearing a spreadsheet hat.

The core identity: QPS × latency = concurrency

The first equation does 90% of the work:

in-flight = QPS × mean latency

A service doing 2,000 QPS at 50ms average latency holds 100 requests in flight at any instant. That number is the floor for every concurrency limit downstream: thread pool, HTTP/2 max streams, database connection pool, gRPC stream count. If your pool holds fewer than 100, requests queue before they even reach the service — the queueing cost just moved.

The reverse direction is the sizing statement: if a node can hold C concurrent requests safely and each takes L seconds, the node's QPS capacity is C/L. A node with a 64-thread pool and 50ms requests tops out at 1,280 QPS in theory — and less in practice, because 100% concurrency is the saturation cliff, not an operating point.

Utilization targets: where you choose to sit on the curve

Queueing theory (M/M/1 and friends) says response time scales as 1/(1−ρ) where ρ is utilization. The practical curve:

• ρ ≤ 0.50: latency is near-idle. Comfortable, and half your money is wasted.
• ρ ≈ 0.70: the knee begins — latency is ~3x idle and the p99 starts moving.
• ρ ≈ 0.85-0.90: mean latency is 6-10x idle; a load spike or a retry storm becomes an incident.
• ρ > 0.90: the cliff. Every extra percent of load adds double-digit latency.

Sensible targets by workload: latency-sensitive user-facing services: 50-70% of measured saturation; batch/queue consumers: 80-85% (they can absorb queueing); stateful systems with failover: lower, because the survivor must absorb the dead node's load.

Headroom: the N+1 arithmetic

The single most common capacity mistake is sizing for today's average. Two corrections:

Peak vs average. Production traffic is a wave: consumer web traffic shows a 2-5x peak-to-average diurnal ratio, retail does 10x+ on Black Friday, and launch-day spikes are unbounded by data. Size from the 99th percentile of daily peaks, not the average, plus a growth multiplier — 2x yearly growth is normal, so a 12-18 month horizon means multiplying today's peak by 1.5-3x.

Failure headroom. If the fleet runs at 90% utilization and a node dies, the survivors run at 90% × N/(N−1) — at N=10 that's 100%: saturation, latency cliff, more timeouts, more retries, more saturation. The N+1 rule: run at a per-node utilization such that losing one node leaves the fleet under the cliff:

per-node target ≤ peak_QPS / (capacity_per_node × (N − 1))

With N=10 nodes and 100 QPS capacity each, 1,000 QPS peak → per-node utilization target = 1000 / (100 × 9) = 0.55. At N=20 the same load allows 0.59 per node — redundancy itself buys headroom.

The worked example

Say the service does 5,000 QPS at peak, mean latency 40ms, and a load test shows a node saturates (queueing begins, p99 doubles) at 500 QPS:

concurrency at 500 QPS:   500 × 0.040 = 20 in-flight per node
required fleet at 60%:    5000 / (500 × 0.60) = 16.7 → 17 nodes
N+1 check at 17 nodes:    5000 / (500 × 16) = 0.625 → fleet survives a loss at 62.5%

The load test is the unskippable step: μ (service rate) cannot be derived from a spec sheet, it must be measured against the real dependency chain — because a service that's really 200 QPS/node, not 500, changes this plan by 2.5x.

import math
 
qps_peak = 5000
capacity_per_node = 500          # measured, not assumed
for target in (0.50, 0.60, 0.70):
    n = math.ceil(qps_peak / (capacity_per_node * target))
    surv = qps_peak / (capacity_per_node * (n - 1))
    print(target, n, f"survivor util {surv:.2f}")

The non-CPU resources

CPU is only one queue. The same identity applies to each:

• Database connections: pool ≥ QPS × connection-hold time. 1,000 QPS with 20ms hold time needs ≥ 20 connections — a pool of 5 was never "slow," it was a queue.
• Memory: RSS per concurrent request (buffers, response, per-connection overhead) × in-flight concurrency + cache/working set.
• Disk: write throughput = QPS × bytes per write; log compaction and retention multiply it.
• Network: QPS × response bytes, plus replication and backup copies (3x+ for multi-region).

Every one of these is the same formula with different units, and the bottleneck is whichever resource hits its cliff first.

The honest closing

Capacity math doesn't tell you the future; it tells you the relationship between your inputs and your failure mode. The arithmetic converts "we should be fine" into "we have 1.9x headroom at 12 months of 40% growth, and a node loss costs us 12% latency at the 99th percentile" — and that sentence is what lets a team sleep before a launch and scale calmly after one.