A strong answer
A strong candidate frames What is eBPF and how do kprobes/kretprobes work? around the exact memory layout, compiler semantics, and hardware execution model covered in eBPF Performance Diagnostics: Kernel Tracepoints, kretprobes, and bpftrace. Explain eBPF bytecode verified and run in the kernel, kprobes on function entry, kretprobes on exit, and user-space tools like bpftrace.
- Zero Overhead, Blind Opacity (Log Files / Top):
- Tells you CPU usage is 100% or that disk latency is high.
- Cannot explain WHY: Which file paths? What query sizes? Which lock was contended?
Consider diagnosing an intermittent tail-latency spike in a storage engine:
* 99.9% of `write()` system calls return in 10 microseconds.
* 0.1% of `write()` calls stall for over **500 milliseconds**.
* Is the stall caused by disk controller write-queue saturation?
* File system journaling contention (`ext4_sync_file`)?
* Or the Linux kernel page-writeback throttle (`balance_dirty_pages`)?
In a production interview, always connect the specification rule to its concrete runtime consequence: state what lives on the stack versus the heap or static storage segment, how the compiler or operating system validates alignments, lifetimes, and error codes, and what happens under adversarial or boundary inputs (such as zero-length buffers, integer overflow, concurrent mutation, or partial I/O). Backing your explanation with a small trace of register/memory state transitions immediately separates senior systems engineers from candidates who only memorized syntax.
## What the interviewer is listening for
- **Mastery of core invariant:** Trace kernel execution using static tracepoints and dynamic kprobes with sub-microsecond overhead.
- **Mastery of core invariant:** Analyze in-kernel safety verification, JIT compilation, and BPF ring buffer maps.
- **Mastery of core invariant:** Capture kernel function latency and return values using kretprobes and hash maps.
- **Hardware and ABI awareness:** Explaining how sizes, alignments, calling conventions, or kernel boundaries behave on real 64-bit targets rather than relying on implementation-defined assumptions.
- **Defensive engineering:** Naming the exact compiler flags (`-Wall -Wextra -Werror`), static analyzers, or runtime sanitizers (`-fsanitize=address,undefined,thread`) that verify correctness in CI.
## Common mistakes
- Explaining only surface syntax without tracing the underlying memory addresses, ownership lifetime, or kernel/user-space state transitions.
- Assuming undefined behavior (such as signed overflow, uninitialized reads, strict-aliasing violations, or dangling pointers) will "just work" because it passed an unoptimized `-O0` debug build.
- Ignoring return codes, partial reads/writes, alignment padding, or cleanup paths when an allocation or syscall fails midway through a function.