The Runtime Theory
mediumApplicationDSA#distributed-systems#consensus#quorum#replication

Why is distributed consensus expensive and when do you need it?

They want the cost model — quorum round trips, fsync on every replica, coordination overhead — plus a sharp line on when correctness demands consensus and when it doesn't.

The Runtime Theory Team2 min readasked at google · cockroachlabs · databricks · amazon

This question is really about the cost model: what consensus buys, what it charges, and whether your problem needs the receipt. A strong answer separates "consensus" from "replication" — they're different bills.

The cost is easy to state precisely. A committed write in Raft must be fsynced on a majority of nodes, and the leader must hear back. On a 5-node cluster with 3-node quorum, that's three fsyncs — each one a disk barrier around 1–10ms on real hardware — plus at least one network round trip to the slowest node in the quorum. Compare a single-node write: one fsync, zero round trips. Consensus is roughly "the slowest majority member + a durable write on every member of it".

text
single-node commit:  fsync(1)                       → ~1-10ms
raft commit (5-node): fsync(3) + max RTT over quorum → 10-50ms in a DC

On top of the per-write bill you pay standing overhead: heartbeats, term bookkeeping, log catch-up, and failover windows where the cluster is read-only. Consensus groups are also hard to move data out of — the log is append-only and replicated, so schema or data migrations touch every replica.

When do you actually need it? The rule of thumb: when two nodes choosing different answers is unrecoverable — a leader that must be unique, a distributed lock, a linearizable read, "which node owns this partition" decisions, or when you need to fail over without losing acknowledged writes. Etcd, ZooKeeper, and Kubernetes control planes are consensus because losing the answer is worse than being briefly unavailable.

When you don't: anything where a stale or arbitrary answer is acceptable. Caches, counters, analytics, event streams — there you want eventual propagation via gossip or anti-entropy, which has zero majority stalls and heals during partitions instead of refusing writes. The real trade is availability-vs-correctness, not speed-vs-speed: consensus makes the system refuse to answer when it can't prove an answer; gossip answers anyway and repairs later.

Edge cases that show depth: read quorums must still intersect write quorums or you get stale linearizable reads; flexible quorums (e.g. 2-of-5 writes, 3-of-5 reads) trade latency for survivability; and "consensus" does not mean agreement about values — it means a single, durable order, which is why the article title matters.

This answer walks

Follow-ups they'll push on

  1. 01How many round trips does a committed write take, end to end?
  2. 02Why is fsync the dominant cost in a quorum commit?
  3. 03When would you pick gossip or anti-entropy replication over a consensus group?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.