The Runtime Theory
hardApplicationDSA#disaster-recovery#rto#rpo#availability

Design disaster recovery with a recovery time objective

Tests whether you can translate a business number into machine decisions — RTO vs. RPO clocks, sync vs. async replication, DNS TTLs, and the rehearsed failover that makes it real.

The Runtime Theory Team2 min readasked at netflix · spotify · amazon · google

This question is testing whether you can translate a business number — "recover in under an hour" — into concrete machine decisions: where data lives, how it replicates, how fast detection is, and what you actually test. A strong answer starts by defining RTO and RPO as two different clocks, then designs backwards from them.

The mental model: RTO and RPO are different clocks. RTO, recovery time objective, is the wall-clock budget from "disaster declared" to "serving traffic." RPO, recovery point objective, is how much data you are allowed to lose, measured in time: an RPO of 15 minutes means you accept losing up to the last 15 minutes of writes. They pull in opposite directions. Synchronous replication shrinks RPO toward zero but adds inter-region RTT to every write — ~75-85 ms for a US-EU sync, so your write latency becomes the network. Asynchronous replication keeps writes fast locally but turns RPO into replication lag: hundreds of ms to seconds under load, unbounded during partitions. Spanner pays the RTT for linearizable cross-region writes; Aurora Global Database keeps writes local and pays with ~1-second failover RPO.

Designing backwards from the RTO. If RTO is 60 minutes, nothing in the plan can take 60 minutes. That means automated detection — health checks with a defined failure budget, not humans paging — automated failover, and DNS TTLs as an enemy: stale answers keep sending traffic to the dead region for 30-300 seconds, and latency-based DNS routing moves herds of users behind a resolver, not individuals. The standby region's capacity must be pre-provisioned: a cold region boots in minutes, so an RTO that can't wait for that requires a warm standby. And the runbook has to be rehearsed — an untested DR plan has an execution time of "unknown."

The data layer is where DR is won. Active-passive with async replication is the honest default: one writer region, the standby holds a replica, RPO is lag-bound. If RPO must be near zero you are in synchronous territory and paying the RTT. Under everything sits the backup floor: versioned object storage at $0.023/GB-month — roughly a tenth of block storage — with 11-nines durability, which is itself a repair-velocity property, not a promise. But a backup only helps if restore has been tested, and restore time is often the forgotten half of RTO. The walkthrough: region fails, health checks declare it, DNS or the global load balancer flips traffic, the standby accepts traffic while replicas catch up or the database fails over, the application starts degraded — read-only if the write path isn't ready.

Tradeoffs and edge cases. Active-passive means the standby costs nearly as much as the active region and serves 0% of users in peacetime; active-active buys peacetime availability but doubles the consistency work — conflicts, split-brain, and per-key ownership become your problem. And the RPO you measure in a drill is the RPO you get: replication lag under drill load is the real number. If the interviewer pushes on testing: failover drills are the only way to know your actual RTO — the first failover should never be the first time the plan works.

This answer walks

Follow-ups they'll push on

  1. 01What happens to RPO when you switch from synchronous to asynchronous replication?
  2. 02Why are DNS TTLs part of your RTO, and what can you do about them?
  3. 03How do you know your RTO is real without a disaster?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.