The Runtime Theory
mediumApplicationDSA#autoscaling#kubernetes#capacity#latency

Design autoscaling that doesn't thrash

Tests whether you understand autoscaling as a latency problem — reaction time vs. provisioning time, hysteresis, cooldown traps, and the scale-down cliff.

The Runtime Theory Team2 min readasked at netflix · spotify · datadog · amazon

This question is testing whether you understand autoscaling as a latency problem with capacity side effects, and whether you know the difference between a stable controller and an oscillator. A strong answer names the two clocks, explains hysteresis, and talks about what happens during the window where new capacity doesn't exist yet.

The mental model: two clocks run simultaneously. Reaction time — metric latency plus collection and decision intervals, often 1-5 minutes of pure delay — and provisioning time — instance creation, image pull, init, readiness, anywhere from 30 seconds to 5+ minutes. For a typical Kubernetes workload: 1 minute of metric lag, 15 seconds of HPA sync, 2 minutes of pod scheduling and readiness, roughly 3 minutes of exposure. During those 3 minutes the existing pods take the full spike — the metric that triggered the scale-out is the one degrading while you wait for the fix.

How the controller decides. HPA runs a fixed control loop every 15 seconds: desiredReplicas = ceil(currentReplicas * utilization / target). At 140% CPU against a 70% target it wants 2x the replicas, and scale-up is effectively one step — the stabilization window for scale-up is zero by default. Scale-down tolerates 5 minutes of low utilization before acting. That asymmetry is the whole point: scale-up races a latency cliff, and scale-down must be lazy, because a downscale that flips into an upscale the next minute is thrash — pods torn down, request queues drained, connection pools reset. Hysteresis, the gap between the threshold that triggers up and the threshold that triggers down, is what turns the controller into a stable system instead of an oscillator.

The cooldown trap. AWS Application Auto Scaling's default 300-second cooldown prevents flapping but means a 10x spike needs four scale events, 5 minutes apart, before the fleet catches up — the p99 has been red for 20 minutes. The fix hierarchy: scale on leading indicators — queue depth, request latency, synthetic probes react before CPU does; oversize the floor, minReplicas is a precomputed latency budget, not a safety net; shrink provisioning time with pre-warmed images, readiness gates that check real dependencies, and fast-start runtimes; and use predictive autoscaling where load is schedulable.

The scale-down cliff. Traffic drops, utilization drops, HPA scales down, a straggler request arrives at a pod being terminated, connection refused, retry storm, utilization spikes, scale up, thrash. The mitigations are graceful and slow termination: terminationGracePeriodSeconds, preStop hooks that drain in-flight requests, and readiness probes that remove pods from the service before the kubelet kills them.

Edge cases. Target 60% if the SLO tolerates 95% — the fleet overshoots the target by the margin of your reaction time. And autoscaling cannot fix a request that arrives before capacity exists: the real design question is whether the old fleet survives the worst-case gap between metric move and new capacity serving. If not, the answer is request shedding, queue depth limits, and latency-aware retry budgets — not autoscaling tuning.

This answer walks

Follow-ups they'll push on

  1. 01Why is scale-up aggressive and scale-down lazy?
  2. 02What are leading indicators, and why is CPU a lagging one?
  3. 03What happens to a request that arrives during the window before new capacity exists?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.