Message Queue Design: Topics, Consumer Groups, and Delivery Guarantees
The mechanics of a message broker, from the moment a producer appends to a partition to the moment a consumer commits an offset. We trace messages through topics, partitions, consumer groups, and replay, and we inject the failures the docs gloss over: a consumer that crashes mid-batch, a broker that loses an ack, and a rebalance that assigns the same message twice. The whole video is about what "at-least-once" actually means on the wire — which side of the broker the duplicates come from, and what you must build on the consumer side to absorb them.
Topics covered:
- Topics, partitions, and where ordering actually holds
- Consumer groups and rebalancing
- Delivery semantics: exactly-once vs the rest
- Producer acks and the data-loss window
- The idempotent consumer pattern
Related articles
Leader-Follower vs P2P Architectures
Quorum write cost, gossip convergence time, and when peer-to-peer wins — CDNs, blockchains, and systems that can live without ordering.
A Load Balancer Is Not a Magic Box
Layer 4 vs layer 7, hashing vs health checks, stickiness, and the failure modes hiding behind the most over-trusted building block in web infrastructure.
Health Checks and Observability
Active vs passive health checks, circuit breaker interplay, and the four concrete reasons a dead node still gets production traffic.
More in System Design
Load Balancers, Explained
Layer 4 vs layer 7, health checks, stickiness, and the failure modes behind the most trusted box in your architecture diagram.
WatchSearch System Design: Inverted Indexes, Tokenization, and Ranking
How a search system actually works — inverted indexes, tokenization, relevance ranking, and the pipeline between a keystroke and ranked results.
DetailsNotification System Design: Fanout, Retries, and Delivery Channels
Designing a notification system — provider abstraction, fanout, retry policies, and how push, email, and SMS channels fail differently.
DetailsChat System Design: WebSockets, Presence, and Message Ordering
Designing a chat system — WebSocket connections, message ordering, presence, and what happens to undelivered messages when a client goes offline.
DetailsRate Limiter Design: Token Bucket, Sliding Window, and Distributed Limits
How rate limiters actually work — token bucket, fixed window, sliding window, and what breaks when the limiter spans multiple machines.
DetailsURL Shortener System Design: Encoding, Storage, and Redirects
Designing a URL shortener — base62 encoding, ID generation, redirect caching, and how the write and read paths differ in scale.
DetailsDatabase Replication Topologies Visualized: Leader, Multi-Leader, and Quorum
How database replication topologies actually work — single leader, multi leader, quorum, and what happens to reads and writes when a node fails.
DetailsCaching Strategies: Cache Aside, Write Through, and Cache Invalidation
How caching strategies actually work in production systems — cache aside, read through, write through, write back, and where stale reads come from.
DetailsLoad Balancer Algorithms: Round Robin, Least Connections, and Hashing
How load balancer algorithms actually distribute requests — round robin, least connections, IP hashing, and the trade-offs each one makes.
DetailsDepth, delivered weekly
One technical dispatch a week — articles and episode notes before they go public.
One technical dispatch per week. No noise.