Two-phase commit (2PC) is the protocol that makes a transaction touching multiple machines behave atomically: either every participant applies it, or none do. It is the machinery behind XA transactions, distributed SQL, and every "distributed transaction" your ORM claims to support. It also has a structural failure mode — blocking — that no amount of tuning removes. Understanding 2PC means understanding both halves: the protocol, and the moment it cannot finish.
The prepare phase
A coordinator (transaction manager) drives the protocol. It sends a PREPARE message to every participant — each shard, database, or queue holding a piece of the transaction. Each participant acquires its locks, writes a "prepared" record to its durable write-ahead log, fsyncs it, and replies.
coordinator:
send PREPARE to all participants
wait for votes (with timeout T)
participant:
lock local resources
append "prepared(tx)" to WAL; fsync
reply YES # promise: can commit, and cannot abort alone
# or reply NO # cannot commit, coordinator must abortThe fsync before the YES is not ceremony. A YES is a promise that the participant will be able to commit this transaction even if it crashes in the next millisecond. That promise is only enforceable if the prepared state is on disk first. A participant that votes YES surrenders the right to abort; a participant that has not yet voted may still abort.
The commit phase
If every participant voted YES, the coordinator appends COMMIT to its own log, then sends COMMIT to each participant. Each participant applies the transaction, releases locks, and replies ACK. Only after all ACKs arrive does the coordinator forget the transaction. A single NO — or a vote lost to a timeout — sends ABORT to everyone.
coordinator:
if all votes YES:
append COMMIT to local log; fsync
send COMMIT to all participants
wait for ACKs
else:
send ABORT to all participantsThree rules keep 2PC correct, and all three are also its fragility:
- A participant that voted
YEScannot change its mind. - The coordinator must not send
COMMITbefore it has all the votes. - A participant in the prepared state holds its locks while it waits for the decision.
Blocking: the failure 2PC cannot survive
The classic failure: the coordinator crashes after collecting YES votes but before sending COMMIT. Every participant that voted YES is now in doubt — it holds locks on a transaction whose outcome it cannot learn. If the coordinator's disk is gone or the machine never returns, that state is permanent: the participants are blocked forever.
No participant can resolve this alone. Aborting would violate the promise it made in prepare; committing could violate atomicity if other participants committed or aborted differently. The only safe action is to wait for a decision that may never arrive. This is the blocking problem, and it is structural: 2PC has exactly one source of decisions, and losing it loses the transaction.
Crash recovery only softens the blow. A recovered coordinator replays its log to find the decision — but only if the decision was logged durably before the crash. A network partition does the same damage: a coordinator that cannot reach a quorum of participants cannot safely decide, so it waits, and so do they.
Why Paxos and Raft replace it
Three-phase commit (3PC) adds a pre-commit state and reduces blocking, but reintroduces it when the network partitions. The real fix used in production is to remove the single point of decision: run the coordinator as a replicated state machine.
Spanner and CockroachDB do exactly this — Paxos or Raft replicates the coordinator's log and each participant's prepared state. The commit decision becomes a consensus decision: a new leader can recover the decision from the replicated log even when the original coordinator is gone, so participants never block on a lost decision. The participant can even ask a quorum "was this transaction committed?" and get a definitive answer.
The honest summary: 2PC is still the right mechanism for atomicity — the prepare/commit handshake, the durable promises, the lock discipline. Consensus is the substrate that makes the decision durable. 2PC without consensus is a protocol whose failure mode is "everything stops"; 2PC over consensus is what distributed databases actually run.
When to avoid it
2PC costs two round trips plus a minimum of three fsyncs per transaction, and it holds locks across machines for the whole window. For high-throughput systems the alternatives are usually better: sagas with compensating actions where atomicity is negotiable, single-writer-per-record designs that need no coordination at all, and idempotent writes where "apply twice" is cheaper than "apply atomically." Choose 2PC when atomicity across machines is non-negotiable — and then choose consensus underneath it, or accept that a coordinator crash can stop your system until a human intervenes.
Production cost model and latency numbers
To apply Two-Phase Commit and Why It Fails effectively in production systems, engineers must move beyond qualitative diagrams and quantify the exact hardware and network costs at each boundary:
| Operation / Boundary | Typical Latency | Dominant Bottleneck | Production Mitigation |
|---|---|---|---|
| L1 / L2 Cache Access | 1 ns – 4 ns | Register dependency chains & pointer chasing | Contiguous memory layouts, cache-line alignment (64 B), and batch iteration |
| Main Memory (DRAM) Fetch | 60 ns – 100 ns | TLB misses, random heap indirection, NUMA hops | HugePages (2 MB), arena/slab allocators, and pre-allocated ring buffers |
| Kernel Syscall / Context Switch | 1 µs – 5 µs | User-to-kernel mode switch, page-table flush, lock contention | Non-blocking readiness loops (epoll / io_uring), batching, and zero-copy I/O |
| NVMe SSD Random Read / fsync | 50 µs – 500 µs | Block I/O queue depth, write amplification, WAL flushes | Group commit, sequential append logs, and bounded buffer-pool pinning |
| Intra-Region RPC Roundtrip | 0.5 ms – 2.5 ms | Serialization, TCP/TLS framing, tail queueing ($L = \lambda W$) | Connection pooling, deadline propagation, and p95 hedged requests |
Failure modes, debugging, and operational checklist
In real incidents, systems rarely fail at average (p50) load—they fail when a slow dependency, skewed hot key, or resource leak pushes concurrency across a non-linear saturation cliff. When reviewing code or architecture that relies on Two-Phase Commit and Why It Fails, walk through three verification questions:
- What is the invariant under partial failure or contention? If a worker crashes mid-operation, a network packet is delayed and reordered, or two threads race on the same state transition, prove which atomic primitive, version number, or idempotency key prevents silent corruption.
- How is backpressure enforced when demand exceeds capacity? Unbounded in-memory queues, unthrottled retries, and blocking calls without explicit deadlines turn a localized
100 msslowdown into a fleet-wide Out-Of-Memory (OOM) cascade. Always bound queue depths, propagate caller deadlines, and shed excess load early. - Which metrics and traces isolate the bottleneck in under 60 seconds? Instrument saturation, queue wait time vs service execution time, cache hit/miss ratios, and error rates by failure reason so on-call engineers can immediately distinguish a compute bottleneck from an I/O or lock-contention stall.