Every message queue worth its dashboard offers "exactly-once delivery." None of them can deliver it, because exactly-once delivery is not a property a queue can have — it is a property a pipeline assembles from parts. What the machine actually does is one of two things: at-most-once (drop or crash and never retry) or at-least-once (retry until acknowledged, duplicates included). Exactly-once is a third thing that exists only in marketing copy and in systems that have built deduplication on top of at-least-once. This article is about that construction.
Where duplicates come from
Duplication is not a bug in the queue; it is the queue behaving correctly under an unreliable network.
- Producer side: a producer sends a message, the network times out, and the producer cannot tell whether the message arrived — so it retries. If the first attempt actually landed, the queue now holds two copies.
- Consumer side: a consumer processes a message, then crashes before acknowledging. The queue redelivers. If the consumer crashed after processing, the same message is processed twice.
Every hop between producer and consumer can duplicate. A queue that "deduplicates" internally only deduplicates its own copies — not the retries your producer makes, and not the redeliveries your consumer triggers.
Consumer offsets: the machine's memory
In Kafka and similar systems, the consumer tracks progress with an offset — the position in the log it has processed. The ordering of two events decides the guarantee:
process(message) → commit(offset) # crash between them → reprocess (at-least-once)
commit(offset) → process(message) # crash between them → message lost (at-most-once)The offset is the consumer's only memory, and it cannot resolve the ambiguity: the consumer cannot atomically "process and remember" across a crash unless processing and remembering happen in the same transaction (more on that below). Offsets make the failure visible; they do not remove it.
The only real mechanism: idempotency keys
The construction that actually delivers exactly-once behavior is at-least-once delivery plus idempotent processing. The consumer must be safe to run twice: re-executing the operation must produce the same result and no double side effect.
The standard tool is the idempotency key: the client generates a unique key per logical operation and sends it with the request; the service records (key, result) and, when a duplicate key arrives, returns the stored result instead of re-executing.
The critical detail — the one that separates working systems from broken ones — is that the idempotency record and the side effect must be committed atomically, in the same transaction:
BEGIN;
INSERT INTO processed (idempotency_key, result)
VALUES (?, ?)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING result;
-- business write(s) go here, in the same transaction
COMMIT;The unique index is the dedupe store. If the dedupe check and the business write are separate operations, a crash between them recreates the exact duplicate you were trying to prevent.
The dedupe store has limits
Idempotency is only as good as the dedupe store's memory:
- Keys must be retained at least as long as the retry horizon. An expired key means a late retry is treated as a new operation — duplicate.
- A crash of the dedupe store loses the record of what ran. If the dedupe store is the same database as the business state, this is survivable; if it is a separate cache, it is not.
- Distributed dedupe stores need their own consensus to avoid two nodes both deciding "first time" for the same key — which is exactly the problem you were trying to solve, one level down.
Kafka's "exactly-once" semantics
Kafka's transactional API (exactly-once semantics, EOS) deserves precision: it uses a transactional coordinator so that producers write batches atomically and consumers can read with read_committed isolation. What it guarantees is within Kafka: the log contains no duplicates, and consumers never see uncommitted batches. The guarantee ends at the Kafka API boundary — Kafka cannot make your database transaction idempotent. End-to-end exactly-once still requires the sink to participate, which means idempotent writes or a transactional outbox.
The outbox pattern
The durable way to close the loop: process the message, write the business state, and commit the consumer offset in one transaction (the transactional outbox). If the transaction commits, the offset moves with the work; if it does not, the message is redelivered — and the idempotency key deduplicates it. Crash recovery is then a replay of a dedupe store, which is a solved problem.
Exactly-once is not a queue feature you buy. It is a pipeline property you build: at-least-once delivery, idempotent handlers, atomic dedupe, and offsets committed with work. When your queue vendor says "exactly-once," ask what happens on the crash between "process" and "ack." That crash is where the guarantee either lives or dies.