The Runtime Theory
easyApplicationDSA#ci-cd#build#containers#deployment

Design a CI/CD pipeline that fails fast

Tests whether you think of a pipeline as a cost curve — cheap gates first, layer-cached builds, and canaries with real rollback — so failures surface in minutes, not in production.

The Runtime Theory Team2 min readasked at netflix · spotify · datadog · amazon

This question is testing whether you think of a pipeline as a cost curve — feedback loops get exponentially more expensive the later they run — and whether you know which checks are cheap, which are slow, and how to order them. A strong answer designs for the cheapest possible failure signal, then walks the stages.

The mental model: order stages by cost-to-fail. A syntax error found in the first minute costs seconds; the same error found after a deploy to production costs an incident. So the pipeline is a gauntlet of increasing cost: the cheap gates run first, and nothing expensive runs before a cheaper gate could have failed.

The walkthrough. Stage one: lint and type-check — milliseconds, no network. Stage two: unit tests on a fast runner, sharded across workers; the rule is each test must take seconds and the whole suite must stay under a few minutes or it stops being run. Stage three: build the container image — and this is where layer order becomes a pipeline design. The Dockerfile must copy package manifests before source so the dependency layer's digest stays stable and the registry deduplicates it; a reordered Dockerfile turns every commit into a full rebuild and re-push of the world. Fail-fast also means never redoing work that already passed: cached layers, cached test results, skipped stages on unchanged paths — a PR that only touches docs shouldn't pay for a full rebuild. Stage four: the deploy, where fail-fast meets the runtime. Smoke test against the real artifact with the real config — the image that will run in prod, not a build-machine approximation. Then canary: 1% of traffic, then 10%, then 100%, with automated rollback on error-rate or latency breach. A canary that doesn't auto-rollback is theater: the pipeline has failed fast only if the rollback path is a tested command, not a fire drill.

Tradeoffs and edge cases. The trap is test-selection overreach: skipping tests to fail fast can let regressions through. The safe version is "fail fast on what changed, run everything on main" — the merge gate is fast, the main branch is thorough. Staging parity is the second trap: a pipeline that deploys to a staging environment materially different from prod fails fast on the wrong things — it validates the pipeline, not the change. And rollback exercises belong in the pipeline itself, not in the incident runbook: if you have never exercised the canary rollback, the pipeline isn't failing fast, it's failing in production. Mention deployment determinism if asked to go deeper: a deploy is only reproducible when the artifact is immutable — same digest, same config, same behavior — which is what the container image gives you.

This answer walks

Follow-ups they'll push on

  1. 01Why does layer order in the Dockerfile matter to CI cost?
  2. 02What makes a canary real rather than theater?
  3. 03Where does fail-fast conflict with test coverage, and how do you resolve it?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.