The Runtime Theory
KernelInternalsmemory

What happens when the machine runs out of memory?

A step-by-step walk from allocation fast-path failure, through kswapd and direct reclaim, the OOM score, victim selection, and SIGKILL — and cgroup v2's gentler cousin.

The Runtime Theory Team3 min read07 steps

layer stack

Kernel

HWHardware
KKernel
RTRuntime
APPApplication
SYSSystem
CLIClient
NETNetwork
TLSCrypto
SRVServer
Go deeperWhat happens when a closure is allocated?What happens when you concatenate strings?What happens when the garbage collector runs?What happens when you malloc(256)?

trace spine

  1. 01 Allocation fast path fails
  2. 02 kswapd wakes; reclaim begins
  3. 03 Direct reclaim when kswapd falls behind
  4. 04 Reclaim fails: OOM condition declared
  5. 05 oom_score computed; victim selected
  6. 06 SIGKILL delivered; pages freed
  7. 07 cgroup v2: scoped OOM instead of system-wide
▸ On this page

"Out of memory" doesn't mean what people think. On Linux, when a process allocates more memory than exists, it usually doesn't fail — something else dies instead. The kernel's memory management has four increasingly desperate stages, and the OOM killer is the last one. Understanding the escalation is how you design services that survive it.

Step 1 — the fast path fails

Every allocation goes through the buddy allocator: per-CPU page lists first, then the buddy system. On modern kernels the allocation is batched with reclaim preemptively (__GFP_* flags, memcg charges), so most allocations never see pressure. But when the free lists are genuinely empty, the allocation enters the slow path with ALLOC_WMARK_MIN and below: the kernel decides to make memory.

Step 2 — kswapd wakes

The kernel's background reclaimer, kswapd (one per NUMA node), wakes and walks the LRU lists. The LRU is split: file pages (page cache — clean ones dropped instantly, dirty ones written back) and anon pages (private memory — only reclaimable by swapping them out). Reclaim order matters: clean file pages first (free, just drop), then dirty file pages (cost: writeback I/O), then anon pages (cost: swap I/O). Under steady pressure this is the normal operating state of a busy server — kswapd churns while everyone else runs.

Step 3 — direct reclaim

If kswapd can't keep up — allocation demand outpaces background reclaim — the allocating process itself stops and reclaims synchronously: shrink_lruvec in its own context, scanning pages, waiting on writeback. This is the famous latency cliff: your malloc or page-fault suddenly takes milliseconds instead of microseconds while it does someone else's memory housekeeping. Direct-reclaim stalls are why steal/stall CPU shows up under memory pressure even with plenty of "free" swap.

Step 4 — the OOM condition

If reclaim still can't satisfy the allocation (no file pages left, anon pages unreclaimable because swap is full or disabled), the kernel declares OOM and calls out_of_memory(). Note the trigger: it's not "memory full" — it's "reclaim failed." A machine with 100GB free can OOM if the allocation can't be satisfied (e.g. vm.overcommit_memory=2 plus a huge single request).

Step 5 — picking the victim

oom_badness() scores every process (within the OOM cgroup scope):

  • Base: RSS + swap usage + page-table memory (roughly "how much would this kill free?").
  • Weighted: root processes slightly favored to survive; forks killed rather than parents (oom_score_adj).
  • Adjusted: /proc/pid/oom_score_adj — -1000 is immortal (init), +1000 volunteers first.

The kernel picks the highest score — the process whose death frees the most memory per "surprise". It's not necessarily the largest process: page tables count, OOM-hung threads count, and children of a process that deliberately oom_score_adj'd itself are penalized (the "if you want to be a watchdog, we'll kill your children" rule).

Step 6 — SIGKILL

The victim gets an unblockable, unhandlable SIGKILL. dmesg records the verdict:

plaintext
Out of memory: Killed process 4123 (mysqld) total-vm:18GB, anon-rss:6.2GB

The kernel then waits for the victim's exit to actually free pages (exiting releases memory via exit_mmap; the killer's only real lever is patience). Until then, every allocation in the system can still stall — the OOM moment is a system-wide event, even with cgroups scoped.

Step 7 — cgroup v2: the civilized version

With cgroup v2, memory limits (memory.max) scope the OOM: the kernel kills a process inside the cgroup (or, with memory.oom.group, the whole group), while the rest of the system keeps running normally. Same machinery, confined blast radius. The memory.high throttle is even gentler — reclaim pressure without death.

What it costs

  • Direct reclaim: ms-scale stalls in the allocating thread, unpredictable jitter.
  • Swap-in/out during reclaim: disk-latency-class (10ms+), amplifies the stall.
  • OOM kill + exit: hundreds of ms before memory is actually released.
  • The quiet cost: kswapd churn under sustained pressure steals CPU/IO that your workload can't see in top.
bash
$ grep -i "oom\|killed" /var/log/kern.log
	Out of memory: Killed process 4123 (mysqld) total-vm:18GB, anon-rss:6.2GB
$ cat /proc/pid/oom_score_adj
	-1000    # why systemd/containers survive your OOMs

The design lesson: the OOM killer is a safety net with bad aim by design — it kills what's cheapest to kill, which is rarely what you'd choose. Run with cgroup limits, set oom_score_adj deliberately, and treat "OOM killed" in logs as a system design failure, not a process failure.

One dispatch a week

If tracing at this altitude interests you, subscribe. One technical explanation per week — no noise, no course spam.

One technical dispatch per week. No noise.