A distributed lock is a lease with teeth. The sequence that makes it safe is not the acquire — it is what happens when the holder dies mid-section. Walk it.
trace stepSystem
The client generates a unique token (UUID, 128 bits) and writes
lock:<name> = <token> with a TTL — in Redlock, 10s, to 5 independent Redis nodes, requiring 3 successes (majority) within a deadline (e.g. 2s). In etcd, the client does a txn that creates the key only if it does not exist, with a lease attached. Two contenders can race, but only one write wins on a majority — the loser retries.trace stepSystem
The key's TTL (or etcd lease) is the lock's entire lifetime. The lock is held only while the key exists. If the holder never renews, the key expires and the lock is free — that expiry is the safety net against a crashed holder.
trace stepSystem
The holder must renew the lease before it lapses: Redlock's rule of thumb is renewal every TTL/3 (3.3s for a 10s lock); etcd clients keepalive every ~1/3 of the lease TTL. Each renewal is a 1ms-scale RTT. If renewals stop — crash, GC pause, network partition — the lock dies with them.
trace stepSystem
Now the client does its work: writes to a storage service, processes a job. The window between last renewal and actual expiry is the danger zone — and the reason the lock alone is never the safety boundary.
trace stepSystem
The holder releases by deleting the key — but only if the value still equals its token (a Lua
if GET == token then DEL on Redis, a compare-and-delete in etcd). This prevents holder A's slow release from deleting holder B's lock, which would be a mutual-exclusion violation, not a bug.trace stepSystem
Here is the mechanism that actually protects data: every write to the guarded resource carries a monotonically increasing fencing token — a sequence number issued at acquire (etcd's
revision, Redlock's clock-derived token). The resource rejects any write whose token is older than the last accepted one. A stale holder — even one that still believes it holds the lock — cannot corrupt state, because its token is frozen.txn: if create(lock/key, token) with lease(TTL=10s):
fencing = revision of that create
while working: keepalive(lease)
write(resource, fencing=fencing) # resource checks fencing > lastSeen
txn: if value == token: delete(lock/key)The cost of all this: each acquire is 3 RTTs (Redlock) or one Raft round trip (etcd), plus a renewal every few seconds for the lease's whole lifetime, plus the fencing check on every guarded write. The safety property it buys — "at most one live lock holder, and stale writers are rejected" — is the standard every job scheduler, leader election, and rate-limited resource is built on.