What are hardware performance counters and what do they measure?

A strong answer

A strong candidate frames What are hardware performance counters and what do they measure? around the exact memory layout, compiler semantics, and hardware execution model covered in Hardware Performance Counters: PMU Profiling with Linux perf and Flame Graphs. Explain PMU counters for cycles, instructions, cache misses, branch misses, and how perf reads them.

While coarse wall-clock measurements tell you that a function is slow, they cannot tell you why it is slow.

• Is the CPU stalling on L3 cache misses?
• Is the pipeline stalled due to branch prediction failures?
• Is the thread descheduled waiting on a lock or I/O?
• Or is the compiler emitting un-vectorized scalar loops?

Historical profiling tools like gprof attempted to solve this by recompiling software with compiler instrumentation (gcc -pg). On every function entry, the compiler injects a trampoline call (mcount).

In a production interview, always connect the specification rule to its concrete runtime consequence: state what lives on the stack versus the heap or static storage segment, how the compiler or operating system validates alignments, lifetimes, and error codes, and what happens under adversarial or boundary inputs (such as zero-length buffers, integer overflow, concurrent mutation, or partial I/O). Backing your explanation with a small trace of register/memory state transitions immediately separates senior systems engineers from candidates who only memorized syntax.

What the interviewer is listening for

• Mastery of core invariant: Profile low-level CPU events (IPC, cache-misses, branch-misses) using Linux perf stat.
• Mastery of core invariant: Sample instruction pointer distribution and call chains using perf record.
• Mastery of core invariant: Differentiate frame pointer, DWARF, and Last Branch Record (LBR) stack unwinding.
• Hardware and ABI awareness: Explaining how sizes, alignments, calling conventions, or kernel boundaries behave on real 64-bit targets rather than relying on implementation-defined assumptions.
• Defensive engineering: Naming the exact compiler flags (-Wall -Wextra -Werror), static analyzers, or runtime sanitizers (-fsanitize=address,undefined,thread) that verify correctness in CI.

Common mistakes

• Explaining only surface syntax without tracing the underlying memory addresses, ownership lifetime, or kernel/user-space state transitions.
• Assuming undefined behavior (such as signed overflow, uninitialized reads, strict-aliasing violations, or dangling pointers) will "just work" because it passed an unoptimized -O0 debug build.
• Ignoring return codes, partial reads/writes, alignment padding, or cleanup paths when an allocation or syscall fails midway through a function.