The OOM Killer and Memory Pressure: Who Dies and Why

Linux would rather kill a process than fail an allocation. That sentence is a design decision with a long history, and it explains the "innocent process was OOM-killed" reports that fill every production postmortem. The machine's logic: a process that cannot be given memory is worse than no process at all. This is how that decision is actually made.

Memory pressure: what "running out" means

"Out of memory" is not "free RAM hit zero." The kernel's job under pressure is reclaim — getting pages back — and it works through a ladder:

1. Drop clean page cache — free instantly, no I/O.
2. Write back and drop dirty pages — costs disk I/O.
3. Shrink slabs — dentries, inodes, and other kernel caches via shrinkers.
4. Swap anonymous pages — if swap exists; the kernel has already balanced this by choosing what to keep in memory.
5. Kill a process — the last rung.

The watermarks drive it: when free memory crosses the low watermark, kswapd starts reclaiming in the background; when it crosses min, the allocating thread is forced into direct reclaim, running the same machinery synchronously — which is why latency spikes right before an OOM. If reclaim cannot keep up with allocation rate, memory pressure turns into a death spiral, and the kernel stops asking nicely.

Overcommit: why malloc lies

malloc() does not allocate memory — it carves address space. Physical pages arrive on first touch (demand paging), so the kernel can let the sum of all allocations exceed physical memory. That's overcommit, controlled by vm.overcommit_memory:

• 0 (default): heuristic — allow everything reasonable, refuse obvious madness.
• 1: always overcommit.
• 2: never — allocations are refused once the sum exceeds RAM + swap.

The bet behind the default: most allocated memory is never touched, and applications that allocate huge pools to use a fraction would break if malloc() failed. Overcommit makes malloc() succeed and moves the failure to the moment of use — where there is no way to say no. Hence: the killer.

The OOM killer's algorithm: oom_score

Every task carries an oom_score from 0–1000. The kernel computes it from a badness heuristic: how much of the machine this process would free (RSS + page tables + swap), with penalties for short runtime, low priority, and cgroup context — the idea being that a process which just started and grabbed half the machine is a better victim than one that has run for a year. Two adjustments matter:

/proc/<pid>/oom_score         # current score, 0-1000
/proc/<pid>/oom_score_adj     # you can shift it: -1000 to +1000
                              # -1000 = immune (OOM_DISABLE)

When the killer fires, it picks the highest-scoring task and sends SIGKILL. If the victim cannot die usefully — e.g., stuck in uninterruptible sleep holding memory — the kernel re-scans and picks the next. Root services get protection via oom_score_adj (systemd sets this for critical units); everything else is judged by the free-memory math. The "innocent" process is innocent in every sense except one: it holds memory the kernel needs.

The kill is logged to the kernel ring buffer — the artifact of every OOM incident:

Out of memory: Killed process 1234 (java) total-vm:42GB, anon-rss:8GB

Cgroup-local killing

With cgroup v2 memory limits, the OOM killer runs inside the container: the kernel chooses a victim within the cgroup that exceeded memory.max. A container OOM kills a container process and logs to memory.events (oom_kill) — without touching the host. This is why a "container crashed" and "host is out of memory" are different incidents with different logs.

Why killing is a feature

The alternatives are worse, and the kernel knows it:

• Failing allocations on demand would surface as ENOMEM or SIGBUS at random points in libraries that never check — a corruption bomb, not an error.
• Thrashing: with swap enabled and no killer, the system burns all CPU swapping pages in and out — the "swap death" hang where nothing runs, ever.
• Retry loops: frameworks that retry failed allocations livelock the machine.

A single fast, logged, externally observable death beats a hang. The contract: fail fast, make it visible (dmesg, oom_score), and let the process supervisor do its job — systemd Restart=always is the standard answer. Userspace killers extend the policy: systemd-oomd and earlyoom kill based on pressure and cgroup before the kernel has to, using context (which cgroup, how long, what's allowed to die) the kernel deliberately doesn't have.

Protecting yourself

• Cap, don't hope: per-service memory.max turns host-wide roulette into per-service policy.
• Watch the pressure: /proc/pressure/memory (PSI) shows some (any waiter) and full (no progress) time percentages — rising full is the pre-OOM signature.
• Tune the score: oom_score_adj for the daemons that must survive; reserve -1000 for the ones that truly cannot restart.
• Swap is a runway, not a shield: it postpones OOM and makes reclaim slow; under pressure, a swapped machine thrashes before it kills.