The Runtime Theory
mediumApplicationDSA#distributed-systems#split-brain#quorum#fencing

What is a split brain and how do you prevent it?

They want the failure-detection impossibility made concrete: two nodes that can't tell 'leader dead' from 'network gone', and the quorum/lease/fencing machinery that prevents divergent writes.

The Runtime Theory Team2 min readasked at cockroachlabs · google · amazon · confluent

Split brain is two or more nodes each believing they are the leader — and both accepting writes — so the system diverges into states that may never converge. The mechanism behind it is the failure-detector problem: a node cannot distinguish "the leader is dead" from "I can't reach the leader". Any timeout-based liveness check is a guess. The interviewer wants to know you understand that the problem is epistemic, not a bug — and that prevention means making "wrong leader" impossible, not detecting it faster.

Prevention has three layers, and a strong answer names all three.

Quorum. Leadership is granted only with a majority of votes — N/2+1. In a 3-node cluster that's 2. If a node can't assemble a majority, it cannot become leader, no matter what it believes. The arithmetic is the guarantee: two majorities of an odd-sized set always intersect, so two leaders cannot both be elected in the same term. The price is explicit: when a 3-node cluster splits 2-1, the minority node steps down and the cluster loses availability on that side — it refuses writes rather than risk divergence. That refusal is the feature.

Lease. Leadership is time-bounded: the leader's authority expires after a lease duration unless renewed by heartbeat. A partitioned leader's claim dies at expiry even though nobody could tell it to stop. Without leases, a reconnecting old leader keeps writing and the two-writer window opens.

Fencing. The storage layer itself rejects stale writers. Every leadership grant carries a monotonically increasing token; every write carries the token; storage accepts a write only if its token is newer than the last accepted one. Even if an old leader's code runs past its lease, its tokens are obsolete and the writes are refused at the disk. This is the layer that makes correctness independent of clocks and timeouts.

Mention the older toolbox: shared quorum disks and STONITH ("shoot the other node in the head") — force-kill the losing side so it physically cannot write. The tradeoff narrative: you cannot have both perfect availability and no split brain, because failure detection is fundamentally uncertain. Every system picks: refuse writes without a majority (safe, briefly unavailable), or accept writes and fence them later (available, needs idempotent recovery). Saying "split brain is what happens when you skip one of quorum, lease, or fencing" is the closing line that lands it.

This answer walks

Follow-ups they'll push on

  1. 01Why can't two nodes in a 3-node cluster both become leader?
  2. 02What does the minority side do while it's partitioned?
  3. 03How does fencing protect the storage layer when the old leader comes back?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.