The Runtime Theory
hardApplicationDSA#networking#operating-systems#tcp#kernel

What actually happens on the server when a TCP packet arrives?

The OS/network boundary question — tests whether you can trace a packet from NIC interrupt to userspace read, and where the actual costs hide.

The Runtime Theory Team2 min readasked at amazon · google · netflix

This is a kernel-walk question — can you name the layers between the NIC and the read() call, and do you know where the real work happens. A strong answer traces the path and flags the expensive steps.

The walkthrough:

  1. NIC → memory. The NIC receives the frame, and (with DMA) writes it into a ring buffer the driver pre-allocated in memory. No CPU involvement yet. The NIC raises a hard interrupt to say "data arrived."
  2. Interrupt → softirq. The CPU's interrupt handler does the minimum — mask the line, mark the device — and defers the heavy work to a softirq (NET_RX_SOFTIRQ). Linux runs softirqs in the context of the interrupted CPU, so receive work happens on the same core that took the interrupt.
  3. Driver → protocol stack. The softirq handler pops the skb (the kernel's socket buffer) from the ring, stamps it with the receive timestamp (that's the NIC timestamp / tcpdump's "arrival time"), and hands it to the IP layer. IP validates checksums, handles reassembly if fragmented, and passes to TCP.
  4. TCP processing. The TCP layer finds the socket from the 4-tuple (hash lookup in the established table), does a checksum validation if not offloaded, handles ACK generation (or schedules delayed ACK, ~40ms later if nothing piggybacks), advances the receive window, and appends the skb to the socket's receive queue.
  5. Wakeup. If a process is blocked in read()/recv(), the kernel marks the socket readable and wakes the task — via the epoll ready list, which wakes the epoll thread, which finds the file descriptor and wakes the waiting worker.
  6. Copy to userspace. The final step is a copy: kernel skb → userspace buffer in the read() syscall. That's a real memory copy of the payload — at 10 Gbps line rate that's ~1.2 GB/s of copying just for receive, which is why zero-copy (io_uring, sendfile, packet mmap) matters.

Where the costs hide: the interrupt itself (mitigated by NAPI — the driver switches to polling for a quota of packets, so one interrupt serves thousands of frames), the per-packet protocol-layer processing (mitigated by GRO, which coalesces adjacent frames into one skb before the stack sees them), and contention — multiple NIC queues and RSS pinning streams to cores so one CPU doesn't serialize everything. On a single-core box, a saturated NIC can burn 100% of one core in softirq before any application code runs.

Edge cases: the connection-teardown path adds timers (TIME_WAIT cleanup, keepalives); SYN floods are handled in the listen queue before any socket exists; and with XDP or AF_XDP the application bypasses the TCP stack entirely and sees raw frames.

This answer walks

Follow-ups they'll push on

  1. 01Where does the kernel actually copy the data?
  2. 02What is receive-side scaling (RSS) and why does it exist?
  3. 03How does a busy-poll or XDP path change this?

More interviews in this topic

One dispatch a week

The trace behind each question, the tradeoff that explains it, and one technical dispatch per week — no noise.

One technical dispatch per week. No noise.