This question tests whether you know the order of operations. The interviewer wants the pressure pipeline — not "the OS crashes."
The mental model: Linux doesn't wait to be empty; memory management is a pressure pipeline. As free memory drops past watermarks, the kernel first reclaims, then swaps, then — as a last resort — kills. Most processes never see the final stage.
Walk through the mechanism. First, overcommit: when a process asks for memory (heap growth, mmap), the kernel usually just reserves virtual space and commits physical pages lazily, on first touch. That's why malloc can succeed and the machine can still run dry — memory is consumed at fault time, not allocation time. The kernel's heuristic (vm.overcommit_memory=0) bets most reservations stay untouched.
When free memory crosses the high watermark, kswapd wakes and reclaims in order of cost: drop clean page-cache pages (free, they're on disk), write back dirty pages, then swap out anonymous pages to the swap device. If allocation pressure outruns kswapd, allocation calls go into direct reclaim — the allocating thread does the reclaim work itself and stalls. That stall is what users feel as a system freeze. If reclaim can't keep up, the kernel runs compaction for huge pages, and finally calls out_of_memory().
The OOM killer then selects a victim: every process carries an oom_score derived mostly from resident memory (RSS plus swap usage), scaled by oom_score_adj, which daemons like systemd set for critical services. The kernel picks the highest score, sends SIGKILL, and reclaims the victim's pages. You see it as the shell reporting exit code 137 and dmesg's "Out of memory: Killed process" line. The system survives; the workload doesn't.
Tradeoffs and edge cases worth naming:
- cgroup memory limits scope the fight: the OOM killer picks from processes inside the offending cgroup, so a container's limit kills the container's processes, not the host's.
- Swap off vs swap on: without swap, anonymous pages are unreclaimable, so the OOM killer fires earlier — swap is a safety valve, not a speed feature.
- OOM vs freeze: with enough overcommit and no swap, the machine can spend minutes in direct reclaim before the killer even runs.
oom_score_adj=-1000pins a process as unkillable — the kernel would rather kill everything else.
A strong closing line: running out of memory is a graduated failure — reclaim first, swap second, kill last — and the OOM killer is the kernel's final ledger: someone's pages must be freed, and the score decides who pays.