How does tail call optimization work and when does it apply?

This is a precision question. The interviewer wants the condition first, then the mechanism, then the caveats. The condition: a call is a tail call when its result is returned directly — the caller has nothing left to do after the call except return. The mechanism: reuse the caller's frame instead of pushing a new one. The caveats: languages, platforms, and ABI constraints.

The mechanism. Normally a call pushes a new frame; the callee's frame sits on top of the caller's, and when the callee returns, ret pops back to the caller. With TCO, the callee runs in the caller's frame: the compiler overwrites argument registers or the caller's argument slots, jumps to the callee instead of calling it, and the callee returns directly to the caller's caller. Effectively the recursion becomes a loop — constant stack depth, no growth:

int sum(int n, int acc) {
    if (n == 0) return acc;
    return sum(n - 1, acc + n);  /* tail call: result returned directly */
}

Without TCO, this recurses n frames deep and stack-overflows around ~10^5 on a typical 8 MB stack. With TCO, sum(1000000, 0) runs in one frame — the same machine code shape as while (n > 0) acc += n--;.

But "when does it apply" is the trap. Three real gates:

• The call must literally be the last operation — not "last line of code". return sum(n - 1) + 1 is not a tail call: the + 1 runs after the call returns. Multiplication, array indexing, finally blocks, and destructors all destroy the tail position.
• The language/compiler must actually do it. C and C++ compilers do it opportunistically (-O2 will usually), but the C standard doesn't require it. The JVM famously does not apply it (stack traces and security-manager era assumptions), though Scala works around it with @tailrec to a loop or a trampoline. JavaScript gets proper TCO only in strict mode and only on some engines. Rust's LLVM does it when the ABI allows.
• The frame must be reclaimable. If the caller's frame holds things the callee might see — destructors, references in registers that would be clobbered — the compiler can't simply drop it.

Edge cases worth naming: mutual recursion (tail calls work across function boundaries, not just self-recursion); exceptions and stack traces — with TCO, a thrown exception shows a truncated stack because frames were reused; and trampolining as the manual fallback: each recursive step returns a thunk, and a loop runs it, forcing constant stack use regardless of compiler support.

A strong closing: "TCO is not a recursion trick; it's the compiler observing that a call is a jump, and making the frame disappear."