A Change Travels Through Delivery and Feedback

Continuous delivery makes a change releasable through repeatable build, validation, and deployment steps. Observability uses emitted signals to infer internal state from outside the system. Together they close a loop: delivery pushes a change out, observability reports whether the push was safe, and the answer decides whether the rollout continues or reverses.

The promise is releasable, not released

The distinction matters more than it sounds. Continuous delivery means every green commit produces an artifact that could go to production on demand; continuous deployment means it actually does. The value of delivery is that the decision to release becomes a cheap, low-ceremony choice rather than an event — because the build, the migration, and the rollback path have all been exercised recently, on real code.

What the pipeline proves is exactly what it is configured to check, and nothing more:

compile + lint          proves the code parses and matches style
unit tests              proves logic against assumed inputs
integration tests       proves the wiring against real dependencies
migration check         proves the schema change is reversible
canary                  proves behavior under real traffic and real data

A green pipeline does not prove the configuration is right, that a migration behaves under production cardinality, or that the feature is what anyone wanted. The deployment step is where those assumptions meet data — which is why the deployment itself has to be structured as a measurement.

Deploy as a controlled experiment

The pattern that separates a safe rollout from a hopeful one: build once, ship the same digest to every environment, put it in front of a small cohort, and compare against a control before widening.

# rollout policy — same digest everywhere, gated by measurement
artifact: registry.example.dev/api@sha256:4f3a…c91e
stages:
  - cohort: "1%"
    observeMinutes: 15
    compareWith: "previous-version, same 15-minute window"
    haltIf:
      errorRateDelta: "+0.3pp"      # percentage points, not percent
      p95LatencyDelta: "+20ms"
  - cohort: "25%"
    observeMinutes: 15
  - cohort: "100%"

Two details make this work in practice. First, the comparison window must be concurrent — comparing against yesterday invites weather (traffic mix, cache warmth, a downstream incident) into the verdict. Second, the cohort needs enough traffic to be statistically meaningful: at 5,000 req/s, 1% is 50 req/s, which is 45,000 requests over 15 minutes — enough to detect a 0.3 percentage-point error lift. At 50 req/s total, the same cohort sees 45,000 requests in twenty-five hours, and a "clean" canary tells you nothing.

Observability: three signals, three questions

Logs explain individual events, metrics show aggregated trends, and traces connect work across service boundaries. Each answers a different question, and none substitutes for the others:

metric : "error rate rose at 14:02"            → when, how much, which fleet
trace  : "the 400 ms was spent in inventory"   → where, in which dependency
log    : "reservation rejected: stale promo"   → why, for this request

Percentiles are where teams most often fool themselves. Take a service where 99% of requests take 20 ms and 1% take 200 ms:

before : 0.99 × 20 ms + 0.01 × 200 ms = 21.8 ms average,  p99 = 200 ms
after  : 0.99 × 20 ms + 0.01 × 400 ms = 23.8 ms average,  p99 = 400 ms

The average moved 9% — easy to miss in a dashboard of averages — while the tail that 1 in 100 users experiences doubled. Alert on the percentile users experience (p95/p99) and on error rate, and always ask for the distribution, not the mean.

Alerts must name an action

An alert is a request for a human to wake up, so the threshold should be tied to user impact or to a budget. The error-budget framing makes that precise: a 99.9% availability SLO gives 43.8 minutes of full outage per month, and alerting on burn rate catches a release that is draining that budget quickly:

# multi-window burn-rate alert (Google SRE style)
- alert: ApiErrorBudgetBurn
  expr: |
    rate(http_requests_total{code=~"5.."}[1h])
      / rate(http_requests_total[1h]) > 0.144
    and
    rate(http_requests_total{code=~"5.."}[5m])
      / rate(http_requests_total[5m]) > 0.144
  for: 2m
  labels: { severity: page }
  annotations:
    summary: "Burning 14.4x budget — 2% of the monthly allowance in one hour."

Two windows (fast and sustained) prevent a single bad scrape from paging and a slow bleed from being ignored. Excessive, non-actionable alerts are worse than none: they train responders to ignore the channel, and the one real page is missed.

Rollback plan: what is reversible

A rollback is a plan only if it names the state that must be undone. Binaries are trivially reversible; schema, cached data, and queued messages are not. The standard answer is to make irreversible steps impossible to need: expand–contract migrations (additive first, destructive changes in a later release, compatible with both versions throughout), feature flags so behavior changes without a deploy, and idempotent consumers so replayed messages after a rollback don't double-apply.

Key takeaways

• Delivery is about reversibility, not speed. Every green commit should yield an artifact that is one command from production and one command from back.
• Deploy is an experiment: same digest everywhere, concurrent control window, cohort big enough to detect the effect you care about (45k requests per 15 minutes at 1% of 5k req/s).
• Percentiles over averages. A tail that doubles can move the average by 9%; instrument with metrics, traces, and logs so you can answer when, where, and why in that order.
• Alert on budget burn with an action attached. 99.9% = 43.8 minutes/month; page on a sustained fast burn, not on a raw error spike.
• Name the irreversible state before you ship. If rollback means losing data, the fix is migration design and flags — not a runbook paragraph.

Google SRE Book: Monitoring Distributed Systems covers alert design; the continuous delivery trace walks one release end to end.