Gossip is how clusters exchange membership and state when no node knows everyone and there is no central registry. A single state change ripples through the cluster in a few seconds. Here is the walk.
trace stepSystem
Node A joins, fails, or updates its metadata. Its local state map now contains
A: {version: 7, alive: true, ...}. Nothing is broadcast — A just holds the new truth locally until the next gossip round.trace stepSystem
Every second (Cassandra's default), A picks a random peer — in Cassandra, 1–3 peers with a 100ms timeout, biased toward nodes it hasn't talked to recently. It sends a digest of its state map: a list of
(node, version) pairs, not the full payloads.trace stepSystem
The peer B replies with the versions it has, and A responds with the full records B is missing (and vice versa). The exchange is symmetric — both nodes send and receive in one round, so one RTT (~0.5–2ms) moves state in both directions.
trace stepSystem
B applies A's records using version numbers: if B has
C: v5 and A says C: v7, B takes v7. If versions conflict (two nodes incremented the same record independently), B keeps both and resolves by a deterministic tie-break — for membership, a higher incarnation number wins, which is how a node that "died" and returned can't be resurrected by a stale rumor.trace stepSystem
Next round, B's digest includes the record it just learned from A. This is the infection step: each round doubles (approximately) the number of nodes that know the state. That exponential spread is the entire point of gossip.
trace stepSystem
After k rounds, the probability a random node hasn't heard the rumor is (1 − fanout/n)^k. With fanout 3, a 100-node cluster converges in ~5 rounds — 5 seconds at a 1s interval; a 10,000-node cluster needs only ~9 rounds. The graph's diameter doesn't matter; the fanout does.
A: pick 1..3 random peers
A: send digest [(A,7), (B,4), (C,9)]
B: reply digest [(A,6), (B,4), (D,2)] # A learns B is behind on A
A: send full record A@v7 # B fills the gap
B: update local map, schedule gossip # next round carries A@v7 onwardThe cost model is the takeaway: one node's change costs O(fanout × log n) messages total and converges in seconds, with no single point of failure and no ordering guarantees. If your system needs a global order, gossip will not give it to you — but for membership, failure detection, and cluster metadata, it is the cheapest self-healing structure that exists.