JIT Compilation: Why Hot Loops Become Machine Code

A JIT compiler watches your program run, finds the parts that execute the most, and compiles those parts to native machine code — at runtime, based on what it has already observed. This is not theoretical. Every modern JavaScript engine, the JVM, .NET's RyuJIT, and LuaJIT do this continuously. The result is that "interpreted" languages can match C for tight loops, because the runtime converts them to C — and sometimes better than C, because the runtime knows things the static compiler cannot.

Why interpreted code is slow

An interpreter executes source code (or bytecode) by walking through instructions one at a time, dispatching each to a handler:

// the bytecode interpreter loop
while (running) {
    instruction = bytecode[pc++];
    switch (instruction) {
        case ADD:    stack[sp-1] += stack[sp]; sp--; break;
        case LOAD:   stack[++sp] = frame[bc[pc++]]; break;
        case CALL:   push_frame(bc[pc++]); break;
        // ... hundreds of cases
    }
}

Every operation goes through the switch. The branch predictor cannot learn the pattern because the instruction stream changes with every function. Cache lines are wasted on the dispatch table. The overhead per operation is 10–100× compared to native code.

The interpreter's one advantage: startup. There is no compilation step. The program runs immediately. For short-lived scripts, this dominates. For anything that runs long enough to matter, the interpreter is a bottleneck.

The JIT's answer: detect, compile, replace

A JIT compiler works in three phases:

1. Detection (profiling)

The interpreter runs normally but instruments certain operations. Every function call, every loop back-edge, every type check records a counter. After a threshold is crossed (typically 1,000–10,000 invocations), the function is flagged as hot.

function add(a, b) {
    return a + b;
}
 
// after 10,000 calls:
//   type feedback: always (int, int)
//   call count: 10,847
//   → compile to machine code

V8's profiler runs on a separate thread, sampling the program state every millisecond. The JVM's profiling happens at method invocation boundaries. Both produce the same output: a list of hot functions with their observed types and branch probabilities.

2. Compilation (optimization)

The compiler takes the hot function and its type feedback, then generates native machine code specialized for the types it has actually seen:

; JavaScript: function add(a, b) { return a + b; }
; after JIT compilation, with type feedback (a, b always int):
 
add:
    mov    eax, edi      ; a (already in edi by ABI)
    add    eax, esi      ; + b (already in esi)
    ret                  ; result in eax

No type checks. No boxing. No dispatch. The compiled code assumes a and b are integers because the profiler observed them to be integers 10,000 times. If they ever aren't, the compiled code must be invalidated — this is deoptimization.

3. Replacement (on-stack replacement)

The hot function is already executing on the stack when its compiled version is ready. The JIT must swap the running interpreted frame for a compiled frame — this is on-stack replacement (OSR). The mechanism is architecture-specific but conceptually simple: the compiler generates a bridge that copies register state from the interpreted frame to the compiled frame's expected locations, then jumps to the compiled code.

After OSR, the function runs at native speed. The interpreter never touches it again.

The three tiers: startup, optimize, optimize harder

Modern JIT compilers use tiered compilation — multiple levels of compilation that trade compilation cost for execution speed:

V8 uses three tiers: Ignition (interpreter), Sparkplug (baseline compiler, no optimization), and TurboFan (full optimizing compiler). The JVM uses C1 (baseline) and C2 (full optimization).

The progression: a function starts interpreted. After ~10 calls, Sparkplug compiles it without optimization — this is fast to compile and already 2–5× faster than interpretation. After ~10,000 calls with stable type feedback, TurboFan compiles it with full optimization — this is slow to compile but produces code that rivals hand-written C.

The reason for tiers: compilation is not free. TurboFan might spend 50ms compiling a function. If that function is called 10,000 times, the 50ms investment pays back in ~5µs per call. If it's called 100 times, the investment never pays back. Tiers let the runtime invest compilation effort only where it will be recouped.

Deoptimization: when the runtime guesses wrong

The JIT compiles based on observed behavior. When that behavior changes — a function that always received integers suddenly receives a string — the compiled code is invalid. The runtime must deoptimize: discard the compiled code and fall back to the interpreter or baseline compiler.

optimized code running
    ↓
type check fails (unexpected type)
    ↓
deoptimization: restore interpreter frame from compiled frame
    ↓
interpreter resumes execution
    ↓
profiler re-evaluates → may recompile with polymorphic path

Deoptimization is expensive — it costs 10–100× more than a normal function call because it involves reconstructing the interpreter's state from scratch. But it is correct: the optimized code would produce wrong results if it continued with the unexpected type.

The practical implication: stable types are fast, unstable types are slow. A function that alternates between two type signatures will constantly deoptimize and recompile, running slower than if it were never JIT-compiled at all.

What the JIT knows that static compilers don't

JIT compilation has a fundamental advantage over ahead-of-time compilation: runtime knowledge. The JIT knows:

• Which branch is actually taken (not which the source code suggests).
• Which types actually appear (not which the type system allows).
• Which function calls are actually inlined (not which the programmer intended).
• What the actual memory layout is (not what the linker guessed).

This means the JIT can optimize based on reality, not prediction. An AOT compiler must generate code for all possible types. A JIT compiler generates code for the types it has actually seen — and that specialization is why JIT-compiled code can outperform AOT code for the same source.

The classic example: in JavaScript, a function add(a, b) that always receives integers compiles to two instructions. The AOT-compiled version must check types on every call. The JIT version skips the checks entirely because it knows — from 10,000 observations — that only integers arrive.

The cost of the JIT itself

JIT compilation consumes CPU. The profiler, the compiler, the deoptimizer — all run on the same cores as your application. In latency-sensitive systems, JIT compilation causes warmup ramps — the first few seconds of a Java or Node.js application are slower than steady-state because the JIT is still compiling hot paths.

This is why benchmarks of JIT-compiled runtimes must include warmup. A 1-second benchmark of Java is measuring interpretation speed. A 30-second benchmark is measuring JIT-compiled speed. They are measuring different things.

The JVM solved this with AOT compilation (GraalVM Native Image) and CDS (Class Data Sharing) — precompiled class metadata that skips interpretation entirely. The trade-off: you lose the JIT's runtime knowledge advantages but gain instant startup.