Every message queue worth its dashboard offers "exactly-once delivery." None of them can deliver it, because exactly-once delivery is not a property a queue can have — it is a property a pipeline assembles from parts. What the machine actually does is one of two things: at-most-once (drop or crash and never retry) or at-least-once (retry until acknowledged, duplicates included). Exactly-once is a third thing that exists only in marketing copy and in systems that have built deduplication on top of at-least-once. This article is about that construction.
Where duplicates come from
Duplication is not a bug in the queue; it is the queue behaving correctly under an unreliable network.
- Producer side: a producer sends a message, the network times out, and the producer cannot tell whether the message arrived — so it retries. If the first attempt actually landed, the queue now holds two copies.
- Consumer side: a consumer processes a message, then crashes before acknowledging. The queue redelivers. If the consumer crashed after processing, the same message is processed twice.
Every hop between producer and consumer can duplicate. A queue that "deduplicates" internally only deduplicates its own copies — not the retries your producer makes, and not the redeliveries your consumer triggers.
Consumer offsets: the machine's memory
In Kafka and similar systems, the consumer tracks progress with an offset — the position in the log it has processed. The ordering of two events decides the guarantee:
process(message) → commit(offset) # crash between them → reprocess (at-least-once)
commit(offset) → process(message) # crash between them → message lost (at-most-once)The offset is the consumer's only memory, and it cannot resolve the ambiguity: the consumer cannot atomically "process and remember" across a crash unless processing and remembering happen in the same transaction (more on that below). Offsets make the failure visible; they do not remove it.
The only real mechanism: idempotency keys
The construction that actually delivers exactly-once behavior is at-least-once delivery plus idempotent processing. The consumer must be safe to run twice: re-executing the operation must produce the same result and no double side effect.
The standard tool is the idempotency key: the client generates a unique key per logical operation and sends it with the request; the service records (key, result) and, when a duplicate key arrives, returns the stored result instead of re-executing.
The critical detail — the one that separates working systems from broken ones — is that the idempotency record and the side effect must be committed atomically, in the same transaction:
BEGIN;
INSERT INTO processed (idempotency_key, result)
VALUES (?, ?)
ON CONFLICT (idempotency_key) DO UPDATE SET idempotency_key = EXCLUDED.idempotency_key
RETURNING result;
-- business write(s) go here, in the same transaction
COMMIT;The unique index is the dedupe store. If the dedupe check and the business write are separate operations, a crash between them recreates the exact duplicate you were trying to prevent.
The dedupe store has limits
Idempotency is only as good as the dedupe store's memory:
- Keys must be retained at least as long as the retry horizon. An expired key means a late retry is treated as a new operation — duplicate.
- A crash of the dedupe store loses the record of what ran. If the dedupe store is the same database as the business state, this is survivable; if it is a separate cache, it is not.
- Distributed dedupe stores need their own consensus to avoid two nodes both deciding "first time" for the same key — which is exactly the problem you were trying to solve, one level down.
Kafka's "exactly-once" semantics
Kafka's transactional API (exactly-once semantics, EOS) deserves precision: it uses a transactional coordinator so that producers write batches atomically and consumers can read with read_committed isolation. What it guarantees is within Kafka: the log contains no duplicates, and consumers never see uncommitted batches. The guarantee ends at the Kafka API boundary — Kafka cannot make your database transaction idempotent. End-to-end exactly-once still requires the sink to participate, which means idempotent writes or a transactional outbox.
The outbox pattern
The durable way to close the loop: process the message, write the business state, and commit the consumer offset in one transaction (the transactional outbox). If the transaction commits, the offset moves with the work; if it does not, the message is redelivered — and the idempotency key deduplicates it. Crash recovery is then a replay of a dedupe store, which is a solved problem.
Exactly-once is not a queue feature you buy. It is a pipeline property you build: at-least-once delivery, idempotent handlers, atomic dedupe, and offsets committed with work. When your queue vendor says "exactly-once," ask what happens on the crash between "process" and "ack." That crash is where the guarantee either lives or dies.
Production cost model and latency numbers
To apply Exactly-Once Delivery Is a Lie effectively in production systems, engineers must move beyond qualitative diagrams and quantify the exact hardware and network costs at each boundary:
| Operation / Boundary | Typical Latency | Dominant Bottleneck | Production Mitigation |
|---|---|---|---|
| L1 / L2 Cache Access | 1 ns – 4 ns | Register dependency chains & pointer chasing | Contiguous memory layouts, cache-line alignment (64 B), and batch iteration |
| Main Memory (DRAM) Fetch | 60 ns – 100 ns | TLB misses, random heap indirection, NUMA hops | HugePages (2 MB), arena/slab allocators, and pre-allocated ring buffers |
| Kernel Syscall / Context Switch | 1 µs – 5 µs | User-to-kernel mode switch, page-table flush, lock contention | Non-blocking readiness loops (epoll / io_uring), batching, and zero-copy I/O |
| NVMe SSD Random Read / fsync | 50 µs – 500 µs | Block I/O queue depth, write amplification, WAL flushes | Group commit, sequential append logs, and bounded buffer-pool pinning |
| Intra-Region RPC Roundtrip | 0.5 ms – 2.5 ms | Serialization, TCP/TLS framing, tail queueing ($L = \lambda W$) | Connection pooling, deadline propagation, and p95 hedged requests |
Failure modes, debugging, and operational checklist
In real incidents, systems rarely fail at average (p50) load—they fail when a slow dependency, skewed hot key, or resource leak pushes concurrency across a non-linear saturation cliff. When reviewing code or architecture that relies on Exactly-Once Delivery Is a Lie, walk through three verification questions:
- What is the invariant under partial failure or contention? If a worker crashes mid-operation, a network packet is delayed and reordered, or two threads race on the same state transition, prove which atomic primitive, version number, or idempotency key prevents silent corruption.
- How is backpressure enforced when demand exceeds capacity? Unbounded in-memory queues, unthrottled retries, and blocking calls without explicit deadlines turn a localized
100 msslowdown into a fleet-wide Out-Of-Memory (OOM) cascade. Always bound queue depths, propagate caller deadlines, and shed excess load early. - Which metrics and traces isolate the bottleneck in under 60 seconds? Instrument saturation, queue wait time vs service execution time, cache hit/miss ratios, and error rates by failure reason so on-call engineers can immediately distinguish a compute bottleneck from an I/O or lock-contention stall.