Replication makes reads scale — and gives every read a chance to be wrong by exactly the amount the replica lags. This trace is about where lag comes from and what it costs.
trace stepSystem
A client inserts a row. The primary validates, writes its WAL (fsync, ~0.5–2ms), applies the change, and returns 200 to the client. No replica was consulted, no ack was required. That is the entire appeal of async replication: commit latency is a single-node cost. The change now exists on one machine.
trace stepSystem
The primary's WAL or binlog shipping thread picks up the change and streams it to replicas — PostgreSQL's wal_sender on a dedicated connection, MySQL's dump thread. Shipping is batched by network and flush behavior: the replica's WAL receiver fsyncs on every
flush (often per transaction) or batches under synchronous_commit = off. A datacenter stream runs at ~1–5ms per batch, but batches can be held for seconds under a busy primary.trace stepSystem
The replica's apply thread replays the change — an index insert, a row update — running the same storage engine work the primary did. Apply throughput is usually slightly lower than primary throughput because it is serial (single-threaded in classic setups) while the primary parallelizes. That asymmetry is the seed of permanent lag under load.
trace stepSystem
The client — or a load balancer — routes the next read to a replica. The replica serves what it has applied: if it is 2 seconds behind, the read is 2 seconds stale. The user's own write is missing. Classic read-your-writes violation: the user sees their data disappear because their read went to a replica that hasn't caught up.
trace stepSystem
Lag is not a mystery metric — it is directly observable:
seconds_behind_master (MySQL), pg_stat_replication.replay_lag (PostgreSQL), and the replica's last-applied WAL position vs the primary's. Healthy clusters sit at 0–100ms; a busy primary with a single-threaded replica can stretch to seconds or minutes, and a stalled replica never converges.trace stepSystem
The standard fixes, in order of cost: (1) route all reads for a session to the primary until the session's last write is confirmed applied on the replica — session pinning; (2) read-your-writes via a version check — the replica answers a read with its apply position and the client retries on the primary if it's behind; (3) bounded staleness (MongoDB maxStalenessSeconds) so reads never exceed a guarantee; (4) synchronous replication for the writes that can't tolerate any lag.
on read from user session:
if lastWriteAppliedOn(replica): return replica.query(...) # replica caught up
else: return primary.query(...) # pay primary costThe economics: async replication gives you near-zero commit latency and reads that scale horizontally, in exchange for reads that can be stale by the replica's lag. The mechanisms that make it usable — session pinning, bounded staleness, and synchronous fallbacks for critical writes — all exist to convert "arbitrary staleness" into "staleness you chose."