The Runtime Theory
hardApplicationDSA#backend#databases#migrations#reliability

How would you migrate a monolith database without downtime?

The full choreography of a zero-downtime migration — dual writes, backfill, read cutover behind a flag, write cutover, and decommission — treated as traffic shaping, not schema work.

The Runtime Theory Team2 min readasked at google · netflix · amazon · shopify

This is the question that separates people who think migrations are a schema task from people who know they're a traffic-shaping task. The interviewer wants the full choreography: you don't move data, you move reads and writes in stages while the old and new systems coexist.

The mental model: every phase is reversible, and the cutover is a ramp, not an event.

Phase one — schema compatibility. The new schema must accept writes the old one can produce, which means expand-migrate-contract: expand with additive changes only (new columns with defaults — never drop or rename in the same step), backfill asynchronously in batches, and contract by dropping the old column only after the new path has proven itself in production.

Phase two — dual writes plus backfill. Every write goes to both stores, with the new side validated but not yet read by the app. The backfill copies history in chunks keyed by an ID or updated-at watermark; it must be idempotent and resumable because it will crash. While it runs, a reconciliation job compares old and new and repairs drift, because dual writes leak — a transaction failing halfway leaves the stores diverged.

Phase three — the read cutover. Move reads to the new store behind a feature flag, ramping 1%, 5%, 50%. This is where the connection pool becomes the plot: the app's pooled connections now point at the new database, and a cold pool hitting an un-warmed database will time out and take the app down — the classic "migration went fine, the app didn't." Warm the new database before traffic arrives: pre-built indexes, WAL pre-warming, a warm cache.

Phase four — write cutover, then decommission. Point writes at the new store, keep the old one read-only for a shadow window — days to weeks, long enough to catch stragglers — and monitor drift and error rates before dropping it.

Tradeoffs and edge cases: the failure mode to design for is divergence, not data loss — a continuous reconcile job is non-negotiable. CDC streaming from old to new (trigger- or log-based) reduces app code changes but adds a live dependency; app-level dual writes are slower to build and easier to reason about. And the most common real-world mistake is testing the cutover at 100% without ever running it at 1% first.

This answer walks

Follow-ups they'll push on

  1. 01Why is the read cutover a percentage-based ramp, not a flip?
  2. 02What does a reconciliation job do, and why must it run continuously?
  3. 03How do you make the backfill resumable and idempotent?
  4. 04When is it safe to drop the old database?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.