Two kinds of I/O exist, and conflating them is the most common performance error in driver land. Control traffic — "start the copy," "what's your status?" — moves over memory-mapped registers. Bulk traffic — the actual data — moves over DMA. One is a few words at CPU speed; the other is megabytes at PCIe speed. Both show up in the same readl/writel family of functions. Here's what each one actually does.
Step 1 — discovering the device
At boot, PCI enumeration reads each function's configuration space and finds the Base Address Registers (BARs): the device tells the kernel "my control registers live at these physical addresses." For a typical NVMe drive: a BAR window of 8-16KB containing the doorbell registers and submission/completion queue pointers.
Step 2 — ioremap
The driver calls ioremap() (or the more constrained ioremap_uc): a kernel virtual mapping to that physical window. Two properties matter:
- Uncached / write-combining: these addresses never enter the CPU cache. A register is a mailbox — caching it means reading stale state. The mapping is marked uncached, so every access goes to the PCIe bus.
- Access rules: 32-bit aligned loads/stores only (
writel/readl); the compiler must not reorder or merge them (READ_ONCE/WRITE_ONCEsemantics are baked into the accessor macros).
Step 3 — the posted write
writel(1, ctrl->doorbell); /* ring the doorbell: "new command!" */An MMIO write becomes a PCIe Memory-Write TLP, sent to the device, and — here's the trick — the CPU doesn't wait. The write is posted: fire-and-forget, ~10-30ns of CPU time. The device processes it whenever it gets to it. This is why drivers can ring doorbells in tight loops: the writes cost almost nothing to issue.
Step 4 — the blocking read
while ((readl(ctrl->status) & READY) == 0) ; /* spin until ready */An MMIO read is a different animal: it's a PCIe Memory-Read TLP, the device must respond with a completion, and the CPU stalls until that response arrives. Round trip over PCIe: ~200-1000ns per read on typical hardware — and if the device is slow (busy firmware), the stall is unbounded. Two practical consequences: (1) drivers minimize register reads and poll them only at coarse intervals; (2) reading a device register inside a hot loop is a latency bomb — 10µs of stalls per 10 reads.
Step 5 — doorbells and the control plane
NVMe's design shows the pattern: the driver writes command descriptors into a shared ring in system memory, then rings a doorbell register (one posted MMIO write) telling the device "commands N-M are ready." The device DMAs the descriptors itself. Batching dozens of commands per doorbell is exactly how modern storage stacks amortize control overhead — one posted write per 64 commands, not per command.
Step 6 — DMA: the bulk path
The actual data never touches MMIO. The driver maps a buffer (or a page-cache page) with dma_map_single, gets a bus address, and hands it to the device in a descriptor. The device DMAs into the buffer directly — no CPU involvement, no cache involvement (DMA is coherent or the driver handles cache sync), at PCIe bandwidth: ~7GB/s per PCIe gen4 lane-quad (x4), ~25-30GB/s for a full x16 slot. For a 64KB NVMe read, the DMA is microseconds; the setup (descriptor, doorbell, interrupt, wakeup) is what the driver spends most of its time on.
Step 7 — completion: MSI-X
When done, the device writes a completion entry into a completion queue (again, DMA into system memory) and raises an MSI-X interrupt. The handler reads the new completion entries (register reads where possible are replaced by memory reads of the DMA'd queue — avoiding step 4's stalls), wakes the waiting I/O, and the I/O returns. With interrupt coalescing the device can delay the interrupt to batch completions — trading latency for interrupt rate.
What it costs
- Posted MMIO write: ~10-30ns CPU time (device processes later).
- MMIO read: ~200-1000ns, CPU stalled the whole time.
- DMA bulk: ~µs per 64KB at device bandwidth — the only sane way to move data.
- MSI-X interrupt handling: ~1-5µs including wakeups.
$ perf stat -e mem-loads,instructions dd if=/dev/nvme0n1 of=/dev/null bs=1M count=1000 2>&1 | grep -E "mem-loads|insns"
1,720,448,310 mem-loads # ~0.6 GB of 1GB went through MMIO; the rest is DMA + page copiesThe ratio to remember: control traffic over MMIO costs ~1000x per byte what bulk traffic costs over DMA. Good drivers minimize MMIO transactions and maximize DMA size; every "why is my driver slow" investigation eventually lands on that balance.