Replication Improves Availability at a Consistency Cost

Replication stores copies of data on multiple machines to improve availability, read capacity, or recovery. The system must define how writes reach replicas and what reads are allowed to observe. "Consistent" is not one universal behavior — the contract should name ordering and freshness guarantees in terms a caller can actually test.

The write path and the lag it creates

The common topology is single-leader: writes go to the leader, which ships a replication stream to followers.

client → leader (WAL) → async ship → follower applies → reader routes here?
 
write acked at t=0; follower applies at t=+120 ms
 
t=0     client UPDATE address, gets "ok"
t=+30ms same client GETs address → routed to follower → old value
        (read-your-writes violated — not an error, a wrong answer)
t=+120ms follower caught up; subsequent reads correct

Asynchronous replication gives you fast writes and the risk of serving stale reads — and of losing acknowledged writes. The loss window is quantified directly:

RPO (data loss on failover) ≈ replication lag × write rate
   lag 1 s   at 500 writes/s  → up to 500 writes vanish if leader dies now
   lag 10 ms at 500 writes/s  → up to 5 writes
   sync to one replica        → 0 acknowledged loss (with W=2 of 3)
                               but every write pays the follower's RTT:
                               +0.5 ms same AZ, +70 ms cross-region

That is the entire sync/async trade in two lines: latency and write availability versus a bounded loss window. Synchronous replication to an unreachable follower blocks writes entirely unless the configuration allows committing without it — so "sync" is a policy with a failure mode, not a safety switch.

Read contracts you can state precisely

Read-your-writes is a contract, and there are three mechanical ways to provide it:

route to the leader            simple, correct, puts all reads on one node
session / affinity stickiness  keep the client on a replica that has applied
                               its writes (works while lag is small)
version constraint             client carries the version/LSN it wrote;
                               replica answers "not yet" or waits until caught up
                               GET /address?after_version=8471
monotonic reads                never move a session to an OLDER replica
                               (prevents back-and-forth flicker between
                               replicas at different lags)

The check question lands exactly here: a user updates an address, then reads from another region and sees the old one. The missing promise is read-your-writes; the fixes above are the menu. Notice that none of them require changing the replication mode — they change routing, which is often cheaper than adding synchronous writes you'll regret under partition.

Lag itself is a number to monitor per replica, not per cluster. A follower that lags 10 ms under normal load can sit at 30 seconds after a long-running ALTER TABLE or a burst of large transactions that the apply thread must chew through — and any read routed to it in that window silently returns stale data. Track replication_lag per node with an alert on both absolute value and rate of change, because the dangerous state is not "laggy" but "lagging faster than it drains."

Quorums: arithmetic, not adjectives

With N replicas, writes acknowledge after W replicas and reads require R replicas. If W + R > N, the write set and read set must intersect — so a read always touches at least one replica holding the newest write:

N=3, W=2, R=2   2+2 > 3 ✓   (intersection ≥ 1)  reads see the latest write
N=3, W=1, R=1   1+1 < 3 ✗   reads may miss writes entirely
N=5, W=3, R=3   3+3 > 5 ✓   tolerates 2 failures, costs 2 extra nodes' writes
 
cost of W=2 of 3: write latency = slowest of the two fastest acks,
                   write availability = needs 2 of 3 nodes up

A quorum is meaningful only with a defined membership (who counts as the five?) and a failure model (crash-stop, not Byzantine). Change either and the arithmetic changes — which is why membership reconfiguration is its own consensus problem, and why "we have quorum" is a statement that must specify quorum of what.

Replicas are not backups

Replication defends against machine failure; it faithfully replicates your mistakes at network speed:

DROP TABLE customers;        → gone on every replica within the lag window
bad migration / bad UPDATE   → applied everywhere, consistently wrong
logical corruption           → copied, byte for byte
 
backup gives you what replication cannot:
  point-in-time recovery (WAL archive + base backup)
  RPO measured in seconds of archived WAL, independent of node count
  a restore you have actually tested — RTO is a rehearsal, not a number

Rule of thumb: replicas for availability, backups for regret — and run a restore drill quarterly, because an untested backup is a hypothesis too.

Key takeaways

• Replication trades a lag window for capacity and availability. Size that window explicitly: RPO ≈ lag × write rate is the number to defend.
• State consistency as a caller-observable contract — read-your-writes, monotonic reads — and implement it with routing or version constraints before reaching for synchronous writes.
• Quorums are intersection arithmetic: W + R > N, with membership and failure model named. Everything else is an adjective.
• Replicas don't replace backups. They scale failure of machines, not failure of judgment.

In Search of an Understandable Consensus Algorithm (Raft) describes the leader-replication log model that consensus-based storage systems like CockroachDB and TiKV implement.