Explain how Raft guarantees Leader Election safety and Log Matching across network partitions, why you should deploy 3 or 5 nodes instead of 4, and why.

A strong answer

A strong candidate frames Explain how Raft guarantees Leader Election safety and Log Matching across network partitions, why you should deploy 3 or 5 nodes instead of 4, and why. around the exact memory layout, compiler semantics, and hardware execution model covered in [Raft Consensus, Majority Quorums ($R + W > N$), and Distributed Lock Fencing Tokens]. Walk through Raft term numbers, the (lastLogTerm, lastLogIndex) election restriction, why a leader only commits entries from its current term by counting replicas, and Martin Kleppmann's GC-pause Fencing Token proof.

Two catastrophic split-brain bugs strike your infrastructure: (1) To save money, an ops team deploys a 4-node etcd/Raft cluster across 2 availability zones (2 nodes in AZ-1, 2 nodes in AZ-2). When the inter-AZ link blips, neither side can form a majority (4/2 + 1 = 3 nodes)—taking down the entire cluster when just 1 AZ (2 nodes) fails! (2) A worker acquires a 10-second Redis distributed lock (SET lock worker1 NX PX 10000), enters a 12-second stop-the-world GC pause, the lock expires at t = 10s, Worker 2 acquires the lock and writes v2 to storage, and at t = 12s Worker 1 wakes up from GC (still believing it holds the lock!) and overwrites v2 with corrupt v1!

When engineers encounter this bottleneck in production, a surface-level API description rarely explains what went wrong. In real workloads, every line of code executes against concrete hardware constraints: fixed-width memory cells, CPU cache lines, kernel descriptor tables, network socket buffers, or accelerator memory buses. Without a clear mental model of Raft Consensus (term, commitIndex), Majority Quorum Intersection ($W + R > N$), and Monotonic Fencing Tokens, debugging degrades into guesswork—tuning timeouts, adding retries, or scaling instances blindly while the underlying architectural flaw remains untouched.

In a production interview, always connect the specification rule to its concrete runtime consequence: state what lives on the stack versus the heap or static storage segment, how the compiler or operating system validates alignments, lifetimes, and error codes, and what happens under adversarial or boundary inputs (such as zero-length buffers, integer overflow, concurrent mutation, or partial I/O). Backing your explanation with a small trace of register/memory state transitions immediately separates senior systems engineers from candidates who only memorized syntax.

What the interviewer is listening for

• Mastery of core invariant: Prove why Majority Quorums ($Q = \lfloor N/2 \rfloor + 1$) prevent split-brain leaders and committed write loss in Raft.
• Mastery of core invariant: Size and place Raft clusters across 3 Availability Zones (1-1-1 or 2-2-1) to survive full AZ outages.
• Mastery of core invariant: Eliminate zombie lock-holder corruption after GC pauses or network delays using Monotonic Fencing Tokens verified by storage.
• Hardware and ABI awareness: Explaining how sizes, alignments, calling conventions, or kernel boundaries behave on real 64-bit targets rather than relying on implementation-defined assumptions.
• Defensive engineering: Naming the exact compiler flags (-Wall -Wextra -Werror), static analyzers, or runtime sanitizers (-fsanitize=address,undefined,thread) that verify correctness in CI.

Common mistakes

• Explaining only surface syntax without tracing the underlying memory addresses, ownership lifetime, or kernel/user-space state transitions.
• Assuming undefined behavior (such as signed overflow, uninitialized reads, strict-aliasing violations, or dangling pointers) will "just work" because it passed an unoptimized -O0 debug build.
• Ignoring return codes, partial reads/writes, alignment padding, or cleanup paths when an allocation or syscall fails midway through a function.