read(fd, buf, 4096) looks like the simplest possible syscall: give me bytes. Behind it is a five-layer pipeline — fd table, page cache, readahead, block layer, and device DMA — and the answer to "how long does it take" is really "which of those layers do you hit?"
Step 1 — resolving the fd
ksys_read → vfs_read. The fd is an index into the process's file descriptor table, pointing at a struct file (which holds the file position, the open flags, and a pointer to the struct file_operations for the underlying filesystem). Every read() acquires and releases that file's lock — under heavy multi-threaded I/O, this lock is a real contention point. ~1-2µs of kernel walking before any data is even considered.
Step 2 — page cache lookup
For a regular file, read_iter → generic_file_read_iter: the kernel computes the pages covering [offset, offset+len) and looks each up in the page cache (the address_space's radix tree / xarray, keyed by (inode, page_index)). The lookup is a lockless RCU read — ~100ns on a hit.
Step 3 — cache hit
Page present and uptodate: mark it accessed (the dirty/accessed bit that feeds the LRU for reclaim), then copy_to_user() — a validated, fault-safe copy into your buffer, ~20-40ns per 4KB. Total: ~1-3µs, no disk involved. This is the number to memorize: cached reads are essentially free, and most workloads never see anything else.
Step 4 — cache miss: readahead engages
Page not in cache (or the mapping flags VERIFY_READ fault handling): the kernel allocates a page and kicks page_cache_ra — readahead. The readahead window doubles on each sequential miss: 16 → 32 → 64 pages... The kernel speculatively submits reads for pages past your request because sequential access is the dominant pattern and latency hides in the pipeline. This is why a 4KB read on a cold cache pulls 128KB from disk.
Step 5 — the bio reaches the block layer
The pages are assembled into a struct bio (the block layer's I/O unit). The block scheduler (mq-deadline on NVMe, none on some devices) reorders and merges it, then the driver (nvme) maps the pages for DMA. Note: readahead submissions get lower priority tags; your synchronous read goes first.
Step 6 — the device and the interrupt
The NVMe controller pulls the data straight into the page-cache pages over PCIe (DMA — no CPU involvement), then raises an MSI-X interrupt. The interrupt handler acknowledges, marks the pages uptodate, and wakes the sleeping read() caller — the waiter transitions from TASK_UNINTERRUPTIBLE to runnable via try_to_wake_up (see the scheduler-wakeup trace). This is the physical minimum: NVMe ~20-80µs, SATA SSD ~100-500µs, HDD ~5-15ms for a cold read.
Step 7 — return
The caller's read() path copies the now-populated pages to user space and returns the byte count. The pages stay in the cache — the next read hits step 3 and costs 1µs instead of 80µs.
What it costs
- Cache hit: ~1-3µs (no I/O).
- Cold NVMe read: ~50-150µs (mostly device latency + queue depth effects).
- Cold HDD read: ~5-15ms (seek + rotation — 1000x the cache hit).
- Readahead in action: a sequential scan of a 1GB file issues ~64KB bios, sustaining near device bandwidth while the CPU spends ~1% of one core.
$ strace -e trace=read -c dd if=/tmp/big.bin of=/dev/null bs=4k count=100000 2>&1 | tail -3
read 100000 0.478127 4 us/call # hot cache: 4µs per call
$ echo 3 > /proc/sys/vm/drop_caches
$ strace -e trace=read -c dd if=/tmp/big.bin of=/dev/null bs=4k count=100000 2>&1 | tail -3
read 100000 145.128190 1451 us/call # cold cache: 1.45ms per callSame syscall, 360x the cost. The read() syscall itself costs ~1µs; the difference is entirely which layer of the pipeline you're asking.