Consider a user updating their profile. The write goes through a leader node, which must persist the change to its write-ahead log, apply it to the in-memory page, replicate to followers, and acknowledge to the client. Each step carries a latency cost and a failure mode.
1. The application sends a write
The service begins a transaction: UPDATE users SET name = 'New Name' WHERE id = 42. This is not the same as committing — the change is buffered, not visible to other transactions yet. The transaction's isolation level (read committed, serializable) determines when other readers see it.
2. The leader appends to the WAL and fsyncs
Before the change is durable, the leader writes the transaction's intent to the write-ahead log and calls fsync. This is a synchronous disk operation — 1-10 ms on most hardware. Until this completes, a crash could lose the transaction entirely. This fsync is the floor for synchronous write latency.
3. The leader applies the change
The WAL entry is applied to the in-memory data page. If the page is full, a page split may occur, which is more expensive. The change is now visible to queries on the leader, even before replication completes.
4. The leader streams the log entry to followers
The leader pushes the WAL record over a streaming replication connection. This is a network hop — fast within a region (1-2 ms), slower across regions (50+ ms). The follower receives and queues the record.
5. The follower applies and acks
The follower writes the record to its own WAL, fsyncs, applies it, and sends an acknowledgment back. Until the follower applies it, reads from that follower will return stale data.
6. Client acknowledgment
If the client requested synchronous replication, the leader waits for at least one follower's ack before confirming commit. If asynchronous, the leader returns immediately and the follower catches up in the background. The choice determines whether a follower lag of 5 seconds means 5 seconds of potentially lost data.
This write path is the other half of the system: the request-through-load-balanced-service trace shows the read path; this trace shows the write path. The deadlock trace shows what happens when transactions conflict.
Hardware, runtime, and observability notes
When tracing Trace a Database Write in a Leader-Follower Setup in a live production environment (subsystem: storage, primary layer: system), keep the following step-level cost model and diagnostic checklist in mind:
- Step 1 (application): The application sends a write transaction — verifies state invariants before advancing the execution pointer.
- Step 2 (system): The leader appends to the WAL and fsyncs — verifies state invariants before advancing the execution pointer.
- Step 3 (system): The leader applies the change to the data page — verifies state invariants before advancing the execution pointer.
- Step 4 (system): The leader streams the log entry to followers — verifies state invariants before advancing the execution pointer.
- Step 5 (system): The follower applies the entry and acks — verifies state invariants before advancing the execution pointer.
- Step 6 (system): The leader acknowledges to the client — verifies state invariants before advancing the execution pointer.
Every transition across the system boundary carries both a direct CPU/memory cost (cache-line fills, descriptor lookups, lock acquisitions, or context switches) and a tail-latency queueing risk under contention. To verify this trace on your own service, capture latency histograms and hardware/kernel counters across each step boundary, compare cold-cache versus warm-cache execution paths, and confirm that error or timeout branches release all acquired descriptors, locks, and buffers cleanly.