The Runtime Theory

Replication Failover Sequence: Heartbeat Loss, Promotion, Fencing, and Rejoin

Step through primary failure and failover — missed heartbeats, freshest-log selection, promotion, client redirect, fencing the old primary, and safe rejoin.

The Runtime Theory Team08 stages

trace / request.md

PRIMARYREPLICATESHEARTBEATS STOPFRESHEST LOG WINSREPLICA PROMOTEDCLIENTSREDIRECTEDOLD PRIMARYFENCEDOLD PRIMARYREJOINSREPLICATIONRESUMES

readyThe primary accepts writes and replicates them to followers — synchronously to a quorum, or asynchronously to the rest. Followers track their replication lag in bytes.

Failover is a race between the system's failure detection and its consistency guarantees. The heartbeat threshold decides when a primary is presumed dead; the election decides who replaces it; and the freshness rule — the most complete log wins — decides who is allowed to. Every step exists to answer one question: which replica has seen the most writes?

The answer matters because promotion is only safe when the new primary holds every committed write. That is what fencing protects in the other direction: when the old primary wakes up, its stale epoch makes its writes inadmissible everywhere, so two primaries can never both be accepted. Rejoin is the reconciliation — the old primary truncates the entries that were never committed and accepts the new leader's log as truth. The sequence is a cycle by design: healthy replication, detected failure, election, fence, rejoin, and back to healthy.