Leader Election in Practice
An election is a race to a single, well-defined checkpoint: a ZAB sequence number, a Raft term, or an atomic lock acquisition. We trace the exact operations each system runs — the proposal, the quorum response, the watch notification — and then confront the harder problem: the old leader is still alive and still writing.
Topics covered:
- What an election actually stores: a durable, total order of leadership
- ZooKeeper's Zab: ephemeral sequence nodes and the watch that fires
- etcd's Raft: how terms prevent two leaders from acting
- Leases and TTLs: why leadership must expire
- Fencing tokens: how the storage layer rejects stale writes
- The split-brain moment and what prevents writes, not just claims
Related articles
Consensus Algorithms Aren't About Agreement
Raft, leader election, and split-brain — what consensus actually solves and what it doesn't.
Leader Election and Heartbeats
How heartbeats, term numbers, leases, and fencing tokens stop a distributed cluster from running two leaders at once.
Eventual Consistency Is a Spectrum
Strong, causal, read-your-writes — the consistency zoo and when each model is the right engineering choice.
More in Distributed Systems
Raft Consensus Explained
Leader election, log replication, and safety — the consensus algorithm that powers etcd, CockroachDB, and TiKV, explained from first principles.
WatchData Partitioning Explained
Range, hash, and hybrid partitioning — where each key's row actually lives, how partitions are balanced, and what happens when one partition becomes a hotspot.
DetailsDistributed Locks and Leases
What a lock actually protects in a distributed system — lease-based locks with expiry, fencing tokens against stale holders, and why client crashes are the hard case.
DetailsGossip Protocols
Membership, failure detection, and state propagation — how nodes exchange information through random peer conversations so the whole cluster converges without a coordinator.
DetailsChaos Engineering Basics
Fault injection with a controlled blast radius — killing nodes, dropping packets, and inducing latency to verify that recovery paths actually work, not just that they exist.
DetailsDistributed Tracing Explained
Trace context propagation, span trees, and sampling — how one request's work is reconstructed across services using trace IDs, span IDs, and parent-child relationships.
DetailsConsistent Hashing Visualized
Watch keys land on a hash ring — what consistent hashing actually does when a cache node dies, and why ring position and virtual nodes determine how many keys move.
DetailsDistributed Transactions Explained
Two-phase commit, prepare and commit phases, and the coordinator failure window — how databases coordinate atomic writes across machines and what happens when a participant crashes.
DetailsRaft Consensus Visualized
See Raft's term clock, randomized leader election, and log replication in motion — what actually happens in etcd, CockroachDB, and TiKV when a server fails or the network splits.
DetailsDepth, delivered weekly
One technical dispatch a week — articles and episode notes before they go public.
One technical dispatch per week. No noise.