The Runtime Theory
RuntimeDSAexecution

What happens when the stack overflows?

A step-by-step walk from runaway recursion to the guard page: the mmap'd thread stack, the red zone, the page fault that becomes SIGSEGV, and the runtime's error.

The Runtime Theory Team1 min read05 steps

layer stack

Runtime

HWHardware
KKernel
RTRuntime
APPApplication
SYSSystem
CLIClient
NETNetwork
TLSCrypto
SRVServer

adjacent altitudes in this subsystem are still being traced

trace spine

  1. 01 Recursion grows frames
  2. 02 Stack meets the guard page
  3. 03 Kernel raises SIGSEGV
  4. 04 Runtime converts it to an exception
  5. 05 The stack is finite by design

Every thread has a stack — on Linux, an anonymous mmap of 8 MB, of which only what you touch is resident. The stack grows downward as frames accumulate, and its upper edge is a trap. When recursion outruns the mapping, the trap fires. This is the trace of that fire.

trace stepApplication

An unbounded recursion — a self-referential parser, a missing base case, a cyclic tree — keeps calling itself. Each call executes the prologue we traced in function-call-trace: a frame of locals and saved registers, a few dozen to a few hundred bytes, pushed per level. A 128-byte frame means 60,000+ levels per MB — runaway recursion reaches the limit in milliseconds. Nothing warns in advance: the stack has no "90% full" interrupt.

trace stepHardware

The stack pointer walks past the mapped region's end — into the guard page (the kernel's VM_GROWSDOWN mapping plus, on glibc, an explicit 4 KB guard with PROT_NONE below it, and the 128-byte red zone that signal handlers need). The next frame's store hits this unmapped page, and the CPU's MMU raises a page fault. The kernel inspects the faulting address: if it's within the stack-growth allowance, the kernel grows the stack silently — but beyond it, the fault is unrecoverable.

trace stepKernel

The kernel delivers SIGSEGV to the faulting thread — a synchronous signal, delivered immediately on that thread's return path, not queued. In raw C, the default disposition kills the process: the core dump captures the runaway stack for gdb/lldb to inspect. This is why an overflowing C program crashes — the hardware fault has become a process death with no intermediate step.

trace stepRuntime

Managed runtimes install their own SIGSEGV handler before user code runs. When the signal arrives, the JVM or the JS engine walks the stack (the same walk the JIT uses for OSR), finds that the faulting address is in the stack region, and converts the fault into a recoverable condition — StackOverflowError in Java, RangeError: Maximum call stack size exceeded in JS. The thread unwinds through the error machinery instead of dying; the process survives; a catch block that's shallow enough can even continue. What was a kernel-level execution fault became a language-level exception.

bash
ulimit -s 8192
python3 -c "def f(): f()
f()"            # RecursionError — Python checks depth itself
gcc -x c -o /tmp/so - <<< 'int f(){return f();} int main(){return f();}'
/tmp/so         # Segmentation fault — no runtime, no check
trace stepKernel

The design constraint, then: the 8 MB default is a reservation, not a guarantee of resident memory — untouched pages cost nothing (the mapping is lazy), so large stacks are cheap until they're deep. But the reservation is fixed per thread, and it multiplies: 1,000 threads × 8 MB of address space = 8 GB of virtual memory per process — fine on 64-bit, fatal on 32-bit or in constrained environments. That's why thread stacks are tunable (ulimit -s, pthread_attr_setstacksize), and why a shallow default can be a footgun for legitimate deep recursion.

What the machine actually does is treat memory as a one-shot fuse: frames burn toward an unmapped page, the MMU trips, the kernel signals, and either the process dies or the runtime catches the fault and turns it into a language error. The entire mechanism — guard page, red zone, signal — costs nothing while the stack is healthy, and exactly one fault when it isn't.