TRT
Sign in
RuntimeDSAexecution

What happens when the stack overflows?

A step-by-step walk from runaway recursion to the guard page: the mmap'd thread stack, the red zone, the page fault that becomes SIGSEGV, and the runtime's error.

The Runtime Theory Team1 min read05 steps

trace spine

  1. 01 Recursion grows frames
  2. 02 Stack meets the guard page
  3. 03 Kernel raises SIGSEGV
  4. 04 Runtime converts it to an exception
  5. 05 The stack is finite by design

Every thread has a stack — on Linux, an anonymous mmap of 8 MB, of which only what you touch is resident. The stack grows downward as frames accumulate, and its upper edge is a trap. When recursion outruns the mapping, the trap fires. This is the trace of that fire.

step 01Application

An unbounded recursion — a self-referential parser, a missing base case, a cyclic tree — keeps calling itself. Each call executes the prologue we traced in function-call-trace: a frame of locals and saved registers, a few dozen to a few hundred bytes, pushed per level. A 128-byte frame means 60,000+ levels per MB — runaway recursion reaches the limit in milliseconds. Nothing warns in advance: the stack has no "90% full" interrupt.

step 02Hardware

The stack pointer walks past the mapped region's end — into the guard page (the kernel's VM_GROWSDOWN mapping plus, on glibc, an explicit 4 KB guard with PROT_NONE below it, and the 128-byte red zone that signal handlers need). The next frame's store hits this unmapped page, and the CPU's MMU raises a page fault. The kernel inspects the faulting address: if it's within the stack-growth allowance, the kernel grows the stack silently — but beyond it, the fault is unrecoverable.

step 03Kernel

The kernel delivers SIGSEGV to the faulting thread — a synchronous signal, delivered immediately on that thread's return path, not queued. In raw C, the default disposition kills the process: the core dump captures the runaway stack for gdb/lldb to inspect. This is why an overflowing C program crashes — the hardware fault has become a process death with no intermediate step.

step 04Runtime

Managed runtimes install their own SIGSEGV handler before user code runs. When the signal arrives, the JVM or the JS engine walks the stack (the same walk the JIT uses for OSR), finds that the faulting address is in the stack region, and converts the fault into a recoverable condition — StackOverflowError in Java, RangeError: Maximum call stack size exceeded in JS. The thread unwinds through the error machinery instead of dying; the process survives; a catch block that's shallow enough can even continue. What was a kernel-level execution fault became a language-level exception.

bash
ulimit -s 8192
python3 -c "def f(): f()
f()"            # RecursionError — Python checks depth itself
gcc -x c -o /tmp/so - <<< 'int f(){return f();} int main(){return f();}'
/tmp/so         # Segmentation fault — no runtime, no check
step 05Kernel

The design constraint, then: the 8 MB default is a reservation, not a guarantee of resident memory — untouched pages cost nothing (the mapping is lazy), so large stacks are cheap until they're deep. But the reservation is fixed per thread, and it multiplies: 1,000 threads × 8 MB of address space = 8 GB of virtual memory per process — fine on 64-bit, fatal on 32-bit or in constrained environments. That's why thread stacks are tunable (ulimit -s, pthread_attr_setstacksize), and why a shallow default can be a footgun for legitimate deep recursion.

What the machine actually does is treat memory as a one-shot fuse: frames burn toward an unmapped page, the MMU trips, the kernel signals, and either the process dies or the runtime catches the fault and turns it into a language error. The entire mechanism — guard page, red zone, signal — costs nothing while the stack is healthy, and exactly one fault when it isn't.

Not started

Sign in to save your learning progress.

Sign in to save