The queue exists for exactly one day a year: the day traffic spikes 10x and everything downstream can't keep up. This trace follows that day — where the messages go, who lags, and what brings the system back.
trace stepSystem
A promotion goes live. Producers that were enqueuing 100 msg/s now push 1,000 msg/s into the queue. Producers are synchronous and cheap — a Kafka producer batch flush or an SQS SendMessage — so they can oversupply freely. Nothing downstream is asked for permission yet.
trace stepSystem
This is the queue doing its one job: decoupling. Kafka holds the burst as unread partitions; SQS holds it as messages with a retention window (4–14 days default). The queue grows at 900 msg/s net. The cost of absorption is memory and disk — and the emerging latency of the oldest message, which is now a queue, not a pipe.
trace stepSystem
The consumer fleet was sized for 100 msg/s — say 4 workers at 25 msg/s each. They process 100 msg/s against 1,000 msg/s inbound. Lag (Kafka:
consumer_lag per partition; SQS: ApproximateNumberOfMessages) grows ~900 msg/s. At minute 5, lag is 270,000 messages: ~45 minutes of backlog at current rates. The oldest message in the queue is now minutes old, and any latency SLA is already broken.trace stepSystem
Lag-triggered autoscaling reacts (KEDA polls every ~30s): 12 more workers spin up over the next 2–3 minutes. Steady-state consumption rises toward 400 msg/s. Note the arithmetic — scaling 4 workers to 16 does not multiply throughput 4x if the bottleneck is the downstream database or API, not CPU. If consumers are I/O-bound on a slow dependency, more workers just adds queueing at the dependency.
trace stepSystem
Now the honest part. If the burst is bigger than the max sustainable consumption (the promotion tops out at 2,000 msg/s, the fleet can do 500), no amount of scaling catches up — lag grows forever. The system must push back upstream: the producer gets throttled (the queue API returns 429/backoff), or newer messages are dropped with a rejection signal, or the queue starts rejecting writes at a configured depth (bounded queue semantics). Backpressure is the admission control the queue can't do by itself.
trace stepSystem
The promotion ends. Producers drop to 100 msg/s, the fleet processes 400 msg/s, and the backlog drains at 300 msg/s net — the 270,000-message backlog clears in ~15 minutes. Consumers then idle down; the queue returns to near-zero depth. The burst is over, nothing was lost (unless backpressure dropped messages), and the system is exactly where it started.
lag = sum(consumer_lag per partition)
target = 100 messages per worker
scale = ceil(lag / target) # poll every 30s, min/max boundsThe whole arc — spike, absorb, lag, scale, throttle, drain — is a conservation equation: inbound rate minus consumption rate, integrated over time, is the queue depth. The mechanisms (bounded queues, lag autoscaling, upstream 429s) all exist to keep that integral from reaching infinity, and the queue's job is to make the equation's failure mode "slow and recoverable" instead of "lost data."