The Runtime Theory

Message Queue Flow: Producer, Broker, Consumer

Trace a message from producer publish through broker append and offset assignment to consumer poll, processing, and offset commit in the partition log.

The Runtime Theory Team08 stages

trace / request.md

PRODUCER SENDSTHE EVENTBROKER APPENDSTO THE LOGOFFSET GIVESIDENTITYPUBLISH CONFIRMSCONSUMER POLLSAT ITS OFFSETRECORD ISPROCESSEDPOSITION ISCOMMITTEDSTREAM MOVESFORWARD

readyThe producer serializes an event and sends it to the broker. With acks=all it waits for replicas to confirm; with acks=1 it waits only for the leader's append. The setting trades latency against durability.

A message queue is a WAL that other processes read. This diagram follows one record from producer to consumer: publish, append to the partition log, offset assignment, poll, process, commit. The log is the whole design — records are never modified, only appended, and their identity is their offset. The producer's durability setting (acks) decides how many copies must exist before the publish returns, and the consumer's commit decides how much work can be lost in a crash.

The ordering of stages is the semantics. The broker's append must come before the ack, or the producer would believe in a message that doesn't exist. The consumer's process must come before its commit, or a crash would lose work silently. Both boundaries — ack and commit — are where the system's delivery guarantees are actually made: at-least-once falls out of committing after processing, and duplicates fall out of retrying publishes. Every guarantee in the queue is a choice about these two boundaries.