Leader election answers "who is leader?"; the lease answers "for how long, and how do we know they stopped?" This trace walks a lease-backed leadership: acquire, renew, expire, take over, fence.
trace stepSystem
The elected leader creates a lease-backed key —
cluster/leader = <node-id> with a TTL of 10s in etcd, or a Kubernetes Lease object with renewTime and a 15s leaseDurationSeconds. Creation is a transactional compare-and-set: it only succeeds while no other live lease exists. The cluster records the holder's identity.trace stepSystem
Every ~3s (etcd's keepalive interval, about TTL/3), the leader renews: an etcd
KeepAlive RPC, or a Kubernetes renewTime update. Each renewal is one Raft round trip. The lease's effective lifetime is "now + TTL" after each renewal — so a healthy leader's lease never gets within a heartbeat of expiring.trace stepSystem
The leader crashes, gets partitioned, or stalls (GC pause, long fsync). Renewals stop. The etcd lease expires after 10s — expiry is decided by the cluster's wall clock, not by any single node. Every follower's watch fires with
key deleted. This is the whole failover trigger: not a health check, not a ping — the lease expiring.trace stepSystem
Two or more followers race to claim leadership. The claim is a transactional compare-and-set: create the key only if it does not exist (or update only if the holder is the expired one). Raft serializes the contest; exactly one follower wins. The loser's txn fails with a conflict and it goes back to watching. Winner side effect: it is now leader with a fresh lease.
trace stepSystem
This is the step that makes the whole design safe. Each leadership is an epoch (etcd: the key's revision; Kubernetes: the holder identity plus renewTime). Every write the leader makes — to the state machine, the storage layer, the job scheduler — carries the epoch. Any component that saw the old epoch rejects writes from the old leader. The partitioned old leader may still think it is leader and keep "writing" — but its epoch is frozen and its writes are rejected everywhere that matters.
trace stepSystem
Clients that cached the old leader's address get connection failures and re-resolve; watches fire; the new leader's first heartbeat or lease write re-establishes it. Total downtime: TTL + takeover RTT — roughly 10–11s in the etcd default, 15s + election in Kubernetes. Everything between expiry and the first new write is a "no leader" period, not a "two leaders" period — as long as the fencing check exists.
on watch(lease expired):
txn: create(cluster/leader, me) if not exists # exactly one wins
if won: epoch = revision of create; start keepalive; start fencing writes
else: resume watchThe sequence — acquire, renew, expire, contest, fence, resume — is the skeleton under Kubernetes controller managers, etcd itself, Kafka's controller, and every "leader + lease" pattern in the wild. The mechanism is small; the discipline of fencing every write is what makes it correct.