Distributed Locks and Leases
A distributed lock is a lease: a key with an expiry that must be renewed. We trace the acquire, renew, and release cycle against a real lock store, then break it — the holder stalls, the lease expires, another client acquires, and the first client wakes up still holding its token.
Topics covered:
- What locks protect: mutual exclusion across machines with network delays
- The lease: acquire, TTL, and the renewal loop
- Expiry vs. revocation: why locks must time out instead of being released
- The stale holder problem and fencing tokens at the storage layer
- Clock assumptions: why lease duration depends on clock skew bounds
- Redlock's claims and the criticism of clock-based expiry
- Practical guidance: when a lease suffices and when you need consensus
Related articles
Leader Election and Heartbeats
How heartbeats, term numbers, leases, and fencing tokens stop a distributed cluster from running two leaders at once.
Eventual Consistency Is a Spectrum
Strong, causal, read-your-writes — the consistency zoo and when each model is the right engineering choice.
Why Distributed Systems Are Hard
The CAP theorem's real meaning, partial failures, and the eight fallacies — why networked code is a different discipline.
More in Distributed Systems
Raft Consensus Explained
Leader election, log replication, and safety — the consensus algorithm that powers etcd, CockroachDB, and TiKV, explained from first principles.
WatchData Partitioning Explained
Range, hash, and hybrid partitioning — where each key's row actually lives, how partitions are balanced, and what happens when one partition becomes a hotspot.
DetailsGossip Protocols
Membership, failure detection, and state propagation — how nodes exchange information through random peer conversations so the whole cluster converges without a coordinator.
DetailsChaos Engineering Basics
Fault injection with a controlled blast radius — killing nodes, dropping packets, and inducing latency to verify that recovery paths actually work, not just that they exist.
DetailsDistributed Tracing Explained
Trace context propagation, span trees, and sampling — how one request's work is reconstructed across services using trace IDs, span IDs, and parent-child relationships.
DetailsLeader Election in Practice
How real systems elect leaders — ZooKeeper's Zab, etcd's Raft, and lease-based locks — plus fencing tokens and why a stale leader must be fenced before it writes.
DetailsConsistent Hashing Visualized
Watch keys land on a hash ring — what consistent hashing actually does when a cache node dies, and why ring position and virtual nodes determine how many keys move.
DetailsDistributed Transactions Explained
Two-phase commit, prepare and commit phases, and the coordinator failure window — how databases coordinate atomic writes across machines and what happens when a participant crashes.
DetailsRaft Consensus Visualized
See Raft's term clock, randomized leader election, and log replication in motion — what actually happens in etcd, CockroachDB, and TiKV when a server fails or the network splits.
DetailsDepth, delivered weekly
One technical dispatch a week — articles and episode notes before they go public.
One technical dispatch per week. No noise.