What Really Happens During a System Call

Every time your program reads a file, sends a packet, or allocates memory, it crosses a boundary that most engineers never see: the system call. The program stops executing user code, transitions into the kernel, the kernel does the work on its behalf, and returns the result. The transition is not a function call — it is a hardware-level mode switch that changes the CPU's privilege level, switches the stack, and enters a different world. Understanding this boundary explains why system calls are expensive, why strace slows your program to a crawl, and why the kernel's design matters to application performance.

The user/kernel boundary

Modern CPUs have at least two privilege levels (called rings on x86):

Ring 3 (user mode):
  - your program runs here
  - cannot access hardware directly
  - cannot execute privileged instructions (in, out, cli, hlt)
  - limited virtual address space
 
Ring 0 (kernel mode):
  - the kernel runs here
  - full access to hardware, memory, and CPU instructions
  - can modify page tables, interrupt controllers, DMA
  - runs on the kernel stack of the calling thread

The boundary exists for isolation. A user process cannot corrupt another process's memory, disable interrupts, or reprogram the DMA controller — because the CPU will trap (raise an exception) if a privileged instruction is attempted in Ring 3. The kernel is the only entity that can execute these instructions, and it does so only on behalf of a process that requested it via a system call.

The mechanism: trap + syscall

On x86-64 Linux, a system call uses the syscall instruction:

user code                    kernel
───────────                  ──────
mov rax, 1       ; syscall number (write)
mov rdi, 1       ; fd (stdout)
mov rsi, buf     ; buffer address
mov rdx, 13      ; length
syscall          ; ← hardware trap
                 ;   1. save rip → rcx
                 ;   2. save rflags → r11
                 ;   3. load kernel stack pointer
                 ;   4. switch to Ring 0
                 ;   5. jump to syscall entry point
                 ;   6. look up sys_call_table[rax]
                 ;   7. call the handler
                 ;   8. return value → rax
                 ;   9. sysret → restore Ring 3
                 ;  10. user code resumes

The syscall instruction does in hardware what a software interrupt used to do: it saves the user instruction pointer, switches to the kernel stack, elevates to Ring 0, and jumps to a fixed entry point. The kernel's entry code looks up the syscall number in sys_call_table and dispatches to the appropriate handler.

The sysret instruction reverses the process: it restores the user instruction pointer, drops back to Ring 3, and resumes user code. The entire transition takes 50–200 nanoseconds on modern hardware.

What the kernel does during a syscall

The kernel's syscall handler runs in the context of the calling thread — same address space (temporarily switched to kernel page tables), same kernel stack, same thread of execution. The handler:

1. Validates arguments. Every pointer from user space is checked with copy_from_user() or access_ok() — the kernel cannot trust user pointers. A bad pointer causes -EFAULT, not a kernel crash.

2. Acquires locks. Most kernel subsystems have per-object locks (inode lock, socket lock, file descriptor table lock). The handler acquires the necessary locks, which can contend under load.

3. Does the work. For read(), this means: check file position, find the page in the page cache, if not present trigger disk I/O, copy data to user buffer. For write(), it's similar but with additional journaling and cache management.

4. Copies data to/from user space. copy_to_user() copies kernel data into the user buffer. This requires a TLB flush for the user-space pages (the kernel was using its own page tables) and a memcpy that crosses the privilege boundary.

5. Returns. The result goes into rax, and sysret resumes user code.

The total cost depends on what the kernel had to do. A getpid() syscall (which reads a field from the current task struct) costs ~50–100 nanoseconds. A read() syscall that hits the page cache costs ~200–500 nanoseconds. A read() that misses the page cache and requires disk I/O costs 5–10 milliseconds.

The hidden costs of syscalls

Three costs are invisible in application profiling:

1. TLB invalidation. The kernel has its own page table entries. When the kernel accesses user-space pages (via copy_to_user()), the TLB may have stale entries. The kernel must flush or invalidate TLB entries for the affected pages, causing TLB misses on subsequent user-space accesses.

2. Context overhead. The kernel must save and restore registers, switch page tables, and update various CPU state (MSRs, debug registers). This overhead is fixed per syscall and cannot be optimized away by the application.

3. Cache pollution. The kernel's code and data evict user-space entries from L1/L2 cache. After a syscall returns, the application's hot data may no longer be in cache. This is why strace is devastating to performance — every syscall pollutes the cache with kernel data.

normal execution:
  user code → user code → user code → user code
  (cache stays warm)
 
with strace (traced syscalls):
  user code → syscall → trace → user code → syscall → trace → ...
  (cache is constantly polluted)

Why syscalls are batched: io_uring and epoll

The overhead of individual syscalls is why modern I/O interfaces batch operations:

epoll avoids the per-fd select()/poll() syscall by maintaining a ready list in the kernel. The application calls epoll_wait() once to get all ready file descriptors — one syscall for many events, instead of one syscall per event.

io_uring takes this further: it creates a shared ring buffer between user space and kernel space. The application submits I/O requests by writing to the submission ring (no syscall), and the kernel completes them by writing to the completion ring (no syscall). For most I/O, there are zero syscalls in the steady state.

traditional I/O:
  read(fd1) → syscall → read(fd2) → syscall → read(fd3) → syscall
  (3 syscalls)
 
io_uring:
  submit: [read(fd1), read(fd2), read(fd3)] → one syscall
  complete: [result1, result2, result3]     → one syscall
  (2 syscalls for 3 operations)

The trend is clear: syscalls are moving from the hot path to the batch path. Modern performance-critical applications minimize syscalls by batching, caching, and using shared-memory interfaces (io_uring, mmap) to avoid the kernel boundary entirely.

mmap: bypassing the read/write syscalls

mmap maps a file directly into the process's address space. Reading from the mapped region triggers page faults instead of read() syscalls — the kernel maps the file's pages into the process's page table, and subsequent reads access the pages directly.

traditional read:
  user buffer → syscall → kernel buffer → copy_to_user → user buffer
  (2 copies, 1 syscall)
 
mmap:
  user buffer maps to file page → page fault (first access) → no further overhead
  (0 copies after initial fault)

The trade-off: mmap triggers page faults for each page (a few hundred nanoseconds each), and the kernel must manage the mapping's lifecycle (munmap, msync). For random access across a large file, mmap can be worse than read() because page faults are more expensive than a single sequential read. For sequential access or repeated reads of the same data, mmap eliminates the copy overhead entirely.

What this means for your code

1. Every syscall costs 50–200 nanoseconds of fixed overhead. If your hot path makes 1000 syscalls per second, that's 50–200 microseconds — negligible. If it makes 100,000, that's 5–20 milliseconds — significant. Count your syscalls with perf stat -e syscalls:sys_enter_*.

2. strace is a debugger, not a monitoring tool. It slows the target by 10–100×. Use it to diagnose, not to observe.

3. Batch I/O operations. Use epoll for network I/O, io_uring for disk I/O, and vectorized reads (readv, preadv) to reduce syscall count.

4. mmap is not always faster. For sequential reads, read() with a large buffer is faster because it avoids page fault overhead. For random access or shared data, mmap eliminates copies.

5. The kernel is not slow — the boundary is. The actual work inside the kernel is efficient. The cost is the transition: TLB flushes, cache pollution, privilege switches. Minimize the number of transitions, not the kernel's work.