The OS Network Stack Receive Path: From NIC to recv()

When you call recv(), the data is already somewhere. The interesting question is how it got there — and the answer is a relay race through the kernel where the baton moves from hardware to a memory ring, into a softirq, through socket queues, and finally into your process. Every hop is a buffer, every buffer has a policy, and the latency of your request is the sum of decisions made in code you never wrote.

Step 1: the NIC and the ring buffer

The NIC has its own small memory and its own DMA engine. Before traffic arrives, the driver has already handed the NIC a ring buffer — a circular array of descriptors, each pointing to a kernel memory region (an skb, socket buffer) pre-allocated for incoming packets. When a frame arrives, the NIC writes it straight into the next free slot via DMA — no CPU involved. Then it raises an interrupt.

wire → NIC DMA → ring buffer (per-queue) → raise IRQ → CPU picks it up

The ring depth is a real knob: ethtool -g eth0 shows it. Too shallow and the NIC drops packets when the CPU is busy (rx_missed in ethtool -S); too deep and you've traded drop latency for memory and cache pressure.

Step 2: NAPI — interrupts, but polite

The naive design — one interrupt per packet — would make the CPU spend its whole life switching contexts; a 10 Gbps line carrying 64-byte packets is 14.8 million interrupts per second. So Linux uses NAPI (New API): the first packet of a burst raises an interrupt, and the driver immediately disables further interrupts and switches to polling — a kernel thread (ksoftirqd) drains the ring in a loop for a bounded budget (net.core.netdev_budget, default 300 packets). When the queue empties, interrupts re-enable.

Step 3: GRO and the merge before the stack

Before the network stack sees them, the driver runs GRO (Generic Receive Offload): adjacent packets of the same flow get merged into one bigger skb — up to 64 KB. One unit to process instead of forty; the TCP layer then splits the super-packet back out. GRO is the single biggest receive-throughput win on modern Linux, and it's why ethtool -K eth0 gro on is the default on virtually every distro.

Step 4: softirq — where the protocol logic runs

The packet is now inside the kernel's protocol machinery, and this runs in softirq context — on the same CPU that handled the IRQ, at a priority between the IRQ handler and normal process scheduling. Concretely, the CPU:

1. Runs the ip_rcv path: checksum verify, header validation, routing lookup.
2. Hands the skb to the TCP/UDP layer: tcp_v4_rcv matches it to the connection's receive queue — the socket's backlog, capped by rmem_max-derived limits.
3. If a process is blocked in recv(), wakes it; otherwise the data waits in the queue until someone reads.

The key property: this all happens on the CPU that took the interrupt. That's why "one core is hot" is the classic single-connection signature — the NIC's RSS (Receive Side Scaling) hashes the 4-tuple to one queue, one IRQ, and therefore one CPU:

$ cat /proc/interrupts | grep eth0   # look at the column distribution
$ ethtool -L eth0 combined 4          # more queues = more CPUs can share the load
$ ethtool -X eth0 equal 4             # spread RSS hash buckets across queues

Step 5: the socket queue and recv()

The packet has now reached the socket's receive queue — but the wait is not over. The memory limit on that queue is shared across all of the socket's buffers (net.ipv4.tcp_rmem), and the kernel applies SO_RCVBUF pressure: if the application reads slowly, the window advertised to the peer shrinks (tcp_adv_win_scale), the peer stops sending, and the queue settles at a stable depth. recv() is then a copy: the kernel moves the data from the skb into your user-space buffer, and the skb is freed.

$ ss -tin | head -1     # rcv_scale / rcv_space show the advertised window
$ netstat -s | grep -i drop   # listen-overruns: queue full, packets dropped

The failure modes are all buffer policies

• listen-overruns: the accept queue is full — connections completed but never accept()ed are dropped. Symptoms: SYN retries at the client, "connection reset" with no error in the app.
• rx_missed / rx_dropped: the NIC ring is full — the CPU couldn't drain it. GRO off, single queue, or a softirq budget that's too small.
• One hot core: RSS hashing pinned your flow to one queue. More queues, or RPS (Receive Packet Steering) to spread softirq work across cores.