cgroups and Container Isolation: Why Docker Is Not a VM

A Docker container shares the host kernel. Everything else — the fake filesystem, the isolated network, the CPU and memory limits — is a combination of two kernel mechanisms. Namespaces change what a process sees. cgroups change what a process gets. Neither is a virtual machine, and the difference between "container" and "VM" is not virtualization at all — it is a policy question about who controls the resources.

The two mechanisms

• Namespaces (pid, mount, net, uts, ipc, user, time): a namespace is a renamed view of kernel state. In a new PID namespace, your process is PID 1 and cannot see other namespaces' processes. Mount namespaces replace the filesystem view — that's your container image. Net namespaces give you your own interfaces and routing tables.
• cgroups: hierarchical resource accounting and control. The kernel charges every page, CPU tick, and task to a cgroup and enforces limits on the group. Docker is namespaces + cgroups + overlayfs + seccomp — the last one restricting syscalls the kernel keeps available to everyone.

Since Linux 4.5+ and unified cgroup v2, everything lives in one hierarchy: /sys/fs/cgroup. Each container is a directory under it.

CPU: shares and quotas are not the same thing

cgroup v2 CPU control has two separate levers:

# proportional share under contention — "shares"
echo 100 > /sys/fs/cgroup/app/cpu.weight        # default
 
# hard cap — quota µs per period µs
echo "50000 100000" > /sys/fs/cgroup/app/cpu.max
# → 50000µs of CPU per 100ms period = half a core, hard

cpu.weight is relative: two busy containers at 100 and 300 split a core 25/75 — but if one is idle, the other gets everything. cpu.max is absolute: 50% of a core, always, even if the machine is empty. The practical difference: weight protects neighbors; quota guarantees your ceiling. Throttling shows up in cpu.stat (nr_throttled) — a container "using 100% CPU" that is throttled at half is a quota problem, not a busy problem.

Memory: where containers meet the page cache

memory.max is the hard limit; memory.high is a soft throttle that makes the kernel reclaim aggressively before the hard limit. When a container hits memory.max, the kernel reclaims its own pages — and if it cannot, the cgroup-local OOM killer picks a victim inside the container, logged in memory.events (oom_kill counter).

The subtle part: file pages (page cache) are shared and accounted to the reading cgroup. A container that reads 10 GB of files and stops shows ~10 GB in memory.current — but that memory is reclaimable, so it is never OOM'd. Watch memory.stat's anon vs. file split, not raw usage:

anon (anonymous, must be reclaimed or killed)  vs  file (cache, free to drop)

This is also why free inside a container lies: it reads host /proc/meminfo values through a namespace the kernel does not fake.

pids.max and the fork-bomb valve

The pids controller bounds task count per cgroup: echo 64 > pids.max. Hitting it makes fork() fail with EAGAIN — the kernel's answer to fork bombs that no VM needs, and the reason a runaway container cannot take down the host's process table.

The freezer

cgroup.freeze stops all tasks in a group atomically — docker pause is this. Unlike per-process SIGSTOP, the freezer handles threads mid-creation and children spawned during freeze, and reports state in cgroup.events (frozen=1). It is the clean way to snapshot a container.

IO and everything else

io.weight and io.max throttle block I/O by bandwidth and IOPS; io.bfq-based or BPF-based policies can prioritize interactive workloads over batch. cgroups v2 also gate memory.swap.max (swap can be denied per-container), cpuset (pin to cores/NUMA nodes), and hugetlb — the full resource surface is there.

Why Docker is not a VM

• One kernel, one trust boundary. A kernel exploit or a host kernel bug affects every container at once; a VM carries its own kernel and its own crash domain.
• No memory ownership. The page cache is global: one container's cache eviction pressure affects another's — cgroups bound usage but not steal of shared cache.
• Near-native performance. No hypervisor, no guest page tables, no vCPU emulation — a container syscall is a normal syscall. That's the upside, and it's why containers replaced VMs for density.
• Compensations are layers, not guarantees: seccomp filters syscalls, user namespaces (2.6.23+) de-privilege root, gVisor replaces the kernel with a user-space boundary, and Kata Containers give up the shared kernel to get back real isolation. Choosing between them is choosing where you want the trust boundary to live.