What happens when you mmap() a gigabyte?

mmap is the system call that treats a file — or pure anonymous memory — as if it were part of your address space. The remarkable thing is that a 1GB mmap is nearly free: it creates no pages at all. Everything real happens later, in the fault path, and the trick is understanding which later.

Step 1 — the syscall

char *p = mmap(NULL, 1 << 30, PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

For a file mapping, the fd is checked against the file size (you can map beyond EOF; touching those pages faults a SIGBUS). Then do_mmap.

Step 2 — find the gap

The kernel must place 1GB of virtual address space. It holds mmap_lock in write mode (exclusive — every concurrent thread's page faults and address-space ops wait here), searches the VMA rb-tree for a hole (with ASLR randomization), and inserts a new VMA record: start, end, flags, file. Total cost: ~10-50µs. Zero pages allocated, zero physical memory touched.

Step 3 — laziness

The mapping is a promise: "these addresses will work when you touch them." Linux allocates nothing at mmap time — not page tables, not PTEs. This is demand paging, and it's why malloc is fast and malloc+touch is slow. The heap itself is just a big anonymous MAP_PRIVATE mapping; pages materialize on fault.

Step 4 — first touch

p[0] = 1 → TLB miss → page-table walk → no PTE → page fault (the whole fault path: see the page-fault trace). handle_mm_fault finds the VMA, sees a read fault first, and for anonymous memory does the zero-page trick: it installs the shared global zero page (read-only) into the PTE — no allocation at all on first read. The allocation happens on first write: COW fault, allocate a zeroed 4KB page, copy the zero page, make it writable. That's ~1-2µs per 4KB page, i.e. roughly ~0.5-1ms per gigabyte as your loop touches it — and it's spread out over time, invisible per-fault.

Step 5 — file-backed faults

For MAP_SHARED of a file, the fault path is the read path: the kernel looks the page up in the page cache (see the file-read trace), and on a miss issues a bio with readahead — the mapping's pages are identical to read()'s pages, just arriving on demand. A dirty page is written back by the flusher threads (dirty ratio thresholds: /proc/sys/vm/dirty_ratio). For MAP_PRIVATE, reads come from the cache too; writes COW onto private pages.

Step 6 — THP: faulting in bulk

With transparent huge pages (default always on most distros for anonymous memory), the fault handler tries to allocate a contiguous 2MB page instead of 4KB: one fault per 2MB region, ~512x fewer faults and far fewer TLB misses. The cost: the 2MB page is kept whole — touching 1 byte pins 2MB, and reclaiming a THP requires splitting it first (thp_reclaim stalls).

Step 7 — munmap: the teardown

munmap unmaps the VMA, walks the PTEs to free (or just drop) the pages, and then — the part people forget — performs a TLB shootdown: an IPI to every CPU that might have cached those translations, flushing them. On big machines with many cores, the shootdown can dominate munmap cost (that's why perf shows smp_call_function_many on map/unmap-heavy workloads).

What it costs

mmap of 1GB: ~10-50µs (VMA insertion only).
Touch 1GB (4KB faults): ~0.5-1ms of scattered fault time + ~1GB of zeroing.
Touch 1GB with THP: ~1-2ms in 2MB chunks — fewer faults, same zeroing.
munmap of 1GB: ~100µs-1ms, mostly page-freeing plus TLB shootdown IPIs.

bash

$ time ./touch 1073741824          # mmap 1GB, then write every page
real  0m0.472s                      # ~470ms: page faults + zeroing dominate
$ strace -c ./touch 2>&1 | tail -4
	mmap        1      0.000021  21 us/call
	munmap      1      0.000082  82 us/call

The syscalls cost 100µs total; the 472ms is all invisible fault machinery. That gap — the huge difference between syscall time and wall time — is the entire point of lazy mapping, and the thing to remember before you benchmark anything that allocates memory.