Leader-Follower vs P2P Architectures

Every distributed system eventually answers one question: who is allowed to write, and in what order? The two families of answers — pick a leader, or let everyone write — are not styles; they are opposite risk postures with measurable costs. The difference shows up in write latency, failover behavior, and the kind of consistency you can promise.

Leader-follower: ordering as a service

A leader-based system (Postgres primary, Raft clusters, ZooKeeper, Kafka controller) designates one node as the only writer. All writes funnel through it; it assigns sequence numbers; followers replicate the log. Ordering is free — the leader is the order — and reads from followers are deterministic snapshots of the log.

The costs are structural:

• Write bottleneck. One node takes every write. Write throughput is bounded by one node's single-threaded log-append speed — tens of thousands of ops/s, not millions — and by the disk's fsync rate.
• Quorum write cost. To tolerate f failures, writes must reach f+1 nodes: a majority (N=3 → 2 acks; N=5 → 3). Each commit is a distributed operation — one round trip to followers in Raft's classic form, two in the worst case, with the slowest follower setting the pace. Same-region that's 1-3ms added to every write; cross-region, 50-300ms. That is the real price of consensus, and it's paid on every write, forever.
• Failover is a downtime window. During a leader election no writes commit. Raft elections are randomized 150-300ms windows (plus heartbeat timeouts, which production configurations often set to 10-30 seconds of unavailability). ZooKeeper's session timeout, 10-30s, defines the same window. The system is read-only during election, then the new leader must catch up on the log before serving writes.
• Writes don't scale with reads. Adding followers scales reads and nothing else. Leader CPU, disk, and network are all inelastic.

P2P and gossip: no leader, eventual convergence

Peer-to-peer systems (Cassandra, Dynamo, BitTorrent, most blockchains, service meshes) let any node accept writes. Ordering becomes a distributed problem, so the systems mostly refuse to solve it: writes are timestamped (last-writer-wins), versioned (vector clocks), or treated as immutable content. Convergence is eventual and probabilistic.

Membership and state spread by gossip: each node periodically picks k random peers (typically 3-6) and exchanges state summaries:

loop every T seconds (T ≈ 1s):
  peer = random_peer(k of n)
  push(local_state_digest, peer)
  merge(peer_state_digest, local_state)
  # failure detection: peer absent for T_max → mark suspect → evict

Convergence is logarithmic: each gossip round roughly doubles the set of nodes that know a fact, so a 10,000-node cluster converges in ~14 rounds — about 14-60 seconds with typical intervals. Failure detection is correspondingly fast (SWIM-based memberlist in Consul detects a dead node in 1-5 seconds, versus a leader system's election timeout). The trade is that no one ever knows the whole truth: reads can see stale state, and concurrent writers to the same key produce conflicts that must be merged or dropped.

The cost comparison, stated honestly

The number to internalize: consensus costs a quorum round trip on every write, and leader failover costs a write outage on every failure. P2P moves those costs into read staleness and conflict resolution — which it pays on every read, silently.

When P2P actually wins

• Content distribution (CDNs, BitTorrent, IPFS). Content is immutable and addressable by hash — no ordering question exists. Any node can serve any chunk; conflicts are impossible because equal hashes mean equal content. P2P is not a compromise here, it's the correct answer.
• Blockchains. Nakamoto consensus is explicitly leaderless and probabilistic: ordering is decided by proof-of-work races, finality is statistical (6 confirmations ≈ certain). The system trades deterministic ordering for liveness under adversarial conditions — a property leader-based consensus cannot offer.
• KV stores with quorum reads (Cassandra): the workload is single-key operations with no cross-key semantics, so the lack of global ordering costs little.
• Membership and service discovery at fleet scale (memberlist/SWIM): gossip detects failures in seconds without a single coordinator to babysit.

When the leader wins

Transactions touching multiple keys, exactly-once-ish semantics, rate limiting and quotas (ordering = fairness), append-only event logs, anything where a client must read its own write at high confidence, and every workload where a 10-30 second write outage is a product incident. If correctness depends on ordering, a leader — with its costs — is the honest architecture.

The decision rule is sharp: need ordering, pay the leader's toll; need availability and can live without ordering, go peer-to-peer. Systems that pretend otherwise are usually just leader-based with the leader hidden.