TCP Provides an Ordered Byte Stream

TCP hands your application a reliable, ordered byte stream and asks nothing of you except patience. There are no messages, no record boundaries, and no delivery receipts for your program — only a pipe that guarantees bytes arrive, in order, exactly once, or the connection fails. Every guarantee in that sentence is paid for with sequence numbers, acknowledgements, timers, and a congestion controller that will happily slow you down to protect the network.

The abstraction: no messages, only bytes

The most common misunderstanding in socket programming is that a write() produces a corresponding read(). It does not. TCP may split one 10-byte write across two segments, or coalesce three writes into one segment, and the receiver's read() returns whatever is in its buffer at that moment — possibly 1 byte of a 500-byte message, possibly four messages at once. IP datagrams and TCP segments carry boundaries; the stream does not.

So the application must impose its own framing:

func writeFrame(w io.Writer, payload []byte) error {
    var hdr [4]byte
    binary.BigEndian.PutUint32(hdr[:], uint32(len(payload)))
    if _, err := w.Write(hdr[:]); err != nil {
        return err
    }
    _, err := w.Write(payload)
    return err
}
 
func readFrame(r io.Reader) ([]byte, error) {
    var hdr [4]byte
    if _, err := io.ReadFull(r, hdr[:]); err != nil { // ReadFull loops until 4 bytes arrive
        return nil, err
    }
    n := binary.BigEndian.Uint32(hdr[:])
    if n > 1<<20 { // a declared length must be bounded or this is a memory bomb
        return nil, errors.New("frame too large")
    }
    buf := make([]byte, n)
    _, err := io.ReadFull(r, buf)
    return buf, err
}

The three framing strategies in the wild are length prefix (above — binary protocols, gRPC-style), delimiter (newline-delimited JSON, HTTP/1.1's header/body split), and fixed size. Whichever you pick, read in a loop: a single read() call returning fewer bytes than you asked for is normal behavior, not an error.

Sequence numbers, ACKs, and what "reliable" means

Every byte carries a 32-bit sequence number. The receiver acknowledges the highest contiguous byte it has received (plus selective acknowledgements for gaps), and the sender retransmits anything unacknowledged after a timer fires. Out-of-order segments are buffered and delivered in order; duplicates are discarded.

Two subtleties matter in practice:

• Wraparound. Sequence numbers are modulo 2^32, so at 1 Gbit/s a connection wraps its sequence space in roughly 34 seconds. That is fast enough that the mechanism designed for it — PAWS (protection against wrapped sequence numbers, using timestamps) — must be on, and it is on modern stacks.
• Acknowledged ≠ processed. A TCP ACK means the kernel has copied bytes into a receive buffer. It does not mean your application read them, understood them, or committed the transaction. write() returning success proves even less: the bytes are in your send buffer. Application-level acknowledgement is a separate, necessary layer.

Setup, teardown, and the states you pay for

The handshake is one round trip: SYN, SYN-ACK, ACK. On a 40 ms link that is 40 ms before the first request byte — which is why connection reuse is such a large win, and why HTTP/2 keeps one connection open for hundreds of requests.

Teardown is where the accounting shows up. After a connection closes, the side that closed first keeps the socket in TIME-WAIT for 2 × MSL — 60 seconds on Linux — so that delayed duplicates from the old connection cannot be mistaken for the new one. A client that opens and closes thousands of short connections to one destination can exhaust its own ephemeral port range (Linux's default 32768–60999 is about 28,000 ports per destination) in well under a minute. The other state to know is half-open: if a peer crashes without sending FIN, your connection sits in ESTAB forever unless a keepalive, an application timeout, or a load balancer's idle timer intervenes. Default TCP keepalive is typically two hours.

Flow control versus congestion control

Two different loops protect two different things:

• Flow control protects the receiver. Its advertised window says how much unread data may be in flight; a receiver that stops reading shrinks the window to zero and stops the sender — a classic stall when an application blocks while holding a connection.
• Congestion control protects the network. The sender maintains a congestion window (cwnd), starts at about 10 MSS (roughly 14.6 KB) after a handshake, doubles it per RTT during slow start until loss or a ceiling appears, then switches to a steady-state algorithm (CUBIC on most Linux defaults, BBR where deployed).

The arithmetic explains throughput ceilings. A path with 100 ms RTT and 1 Gbit/s of capacity has a bandwidth-delay product of 12.5 MB: you must have 12.5 MB of unacknowledged data in flight to saturate it. If your congestion window, your socket buffer, or your application's write pattern caps in-flight data at 64 KB, your throughput is 64 KB / 100 ms ≈ 5.1 Mbit/s — 0.5% of the link, with zero packet loss and a perfectly healthy connection.

Loss, retransmission, and timers

Fast retransmit fires after three duplicate ACKs — roughly one RTT after the loss. If there are not enough duplicate ACKs (the last segment in flight, typically), the sender waits for the retransmission timeout, which Linux starts at 1 second and floors at 200 ms. A single loss at the tail of a transfer therefore costs about one RTO, not one RTT: on an idle or low-rate connection, one lost packet is a 200–1,000 ms stall. SACK (selective acknowledgement) lets the receiver describe holes precisely so multiple losses in one window cost one recovery instead of several.

Where this bites in production

• write() succeeded is not "delivered." Without an application-level ack, a dropped connection after a successful write silently loses data — this is why databases log before acking and why money movement needs idempotency keys.
• Stalls without errors. A zero window (slow reader), a lost final segment (waiting on RTO), or a half-open connection after a peer crash all present as "the connection is open but nothing is happening." ss -ti shows the send queue, window, and retransmit counters that identify which one you have.
• Tune the buffers to the path. A high-BDP link needs net.ipv4.tcp_rmem/tcp_wmem (or socket buffer options) large enough to hold the BDP; a 100 ms path with 64 KB buffers is a 5 Mbit/s link pretending to be a gigabit one.
• Framing bugs look like protocol bugs. A parser that assumes one read equals one message works in testing with tiny payloads and fails in production under load, when segmentation changes. If your protocol breaks only under throughput, it is framing.

Further reading: RFC 9293: Transmission Control Protocol.