Backpressure and Flow Control

Backpressure answers a question every system eventually asks: what happens when the producer is faster than the consumer? The naive answer — put the excess in a queue — is a latency landmine. The queue is not free storage; every item in it is a promise that will eventually cost latency, memory, or both.

Unbounded queues: the memory-leak-shaped latency bug

An unbounded queue (a list "it'll be fine", a broker with no max, a thread pool with an unlimited queue) behaves in three stages:

1. Latency grows linearly. A request's wait time is queued requests ahead × service time: 10,000 queued at 10ms each means the last one waits 100 seconds. The p99 — what customers actually feel — goes from 10ms to seconds while the mean stays "fine."
2. Memory grows until something dies. 10k requests × 1KB = 10MB is survivable; 10M × 1KB = 10GB is an OOM. Growth is bounded by whatever the OS kills first.
3. The crash is the cascade's starting gun. OOM → restart → in-flight work dies → clients retry → retries hit the still-overloaded downstream → it queues again. The unbounded queue converts a slow downstream into a fleet-wide outage with a single, obvious mechanism.

The systemic point: an unbounded queue has hidden length — nobody monitors it, and its cost appears in latency percentiles no one looks at until the incident.

Bounded queues: fail fast, or block carefully

The bounded queue is the correction: capacity C, and when full, new arrivals are either rejected or the producer blocks.

Rejection (the default that scales): when the queue is full, return 503 (or drop the connection) immediately. The producer — a load balancer, a client, another service — sees a fast, honest failure and can retry with backoff. The system's latency stays flat: accepted requests experience service time plus at most C × service time of queueing, and the excess is rejected in microseconds instead of queued for seconds.

Blocking (fine within a process, dangerous across one): the producer waits for space. In a thread pool this is bounded concurrency with built-in throttling — but it inverts the failure: producers stall, thread stacks pile up, and a chain of services blocking on each other is a deadlock machine with timeouts for victims.

A rejection policy is a one-line confession of what the system is allowed to do under load — and the queue capacity is the real SLO:

new ThreadPoolExecutor(
    corePoolSize, maxPoolSize, 30, TimeUnit.SECONDS,
    new ArrayBlockingQueue<>(256),          // bounded: the latency budget
    new ThreadPoolExecutor.AbortPolicy());  // reject when full

256 slots at 10ms service time = 2.56 seconds of worst-case queued latency, then fast failure. That's a concrete, testable latency promise — the unbounded version has no equivalent number.

TCP: the master class in flow control

The Internet solved this exact problem twice. TCP has two separate mechanisms:

• Flow control protects the receiver: the receiver advertises a receive window in every ACK — "I have W bytes of buffer, do not send more" — and the sender never exceeds it. That is a bounded queue with rejection at packet granularity, and it's why a slow reader doesn't flood the sender's memory.
• Congestion control protects the network path: the sender runs its own window (cwnd), growing it on success (additive increase) and cutting it in half on loss (multiplicative decrease) — AIMD. Loss is the path's "queue full" signal.

HTTP/2 and QUIC carry the same idea to the application layer with per-stream windows. Every one of these is the same principle: bounded in-flight work, the limit visible to the producer, and overflow turned into feedback rather than buffering.

Kafka lag and the queue you can't see

Message brokers make the queue explicit, but the broker's queue is unbounded by design, so consumers define backpressure by their lag. Lag = queue depth, and its costs are the unbounded-queue ones — replay delay on failover, memory and disk pressure — with one extra: lag hides the problem from the producer, which keeps producing at full rate into a consumer weeks behind. Alerting on consumer lag is alerting on an unbounded queue; the fix is consumer autoscaling or producer throttling, not a bigger broker.

What to actually do

1. Bound every queue in the request path; make the bound a multiple of the latency budget (C × service_time ≤ acceptable p99).
2. Prefer rejection with a clear error (503 + Retry-After) over blocking, across process boundaries, always.
3. Apply backpressure at the edges: the load balancer is the first queue — its connection limit is the outer bound of your whole system.
4. Treat queue depth as a first-class SLO metric. A queue that grows in production is backpressure that hasn't decided its failure mode yet — left undecided, it chooses OOM.

Backpressure is not an optimization; it is the difference between a system that degrades and a system that dies. Bounded queues choose degradation, explicitly and on the spot.