Message Queues: Decoupling the Hard Part

Message queues decouple producers from consumers, enabling asynchronous processing, load leveling, and fault tolerance. What the abstraction hides — delivery semantics, ordering, backpressure, and poison messages — is where production incidents come from. The choice between Kafka, RabbitMQ, and SQS is really a choice about which of those four problems you would rather own.

What a queue actually decouples

Three things, and only three: time (producer and consumer don't need to be up simultaneously), rate (a queue absorbs bursts so consumers can work at their own pace), and availability (a down consumer doesn't fail the producer's request). It does not decouple contracts — the event schema is a distributed API with all the versioning pain of one — and it does not give you consistency: the moment a payment and an event must commit together, you are back at a transaction (an outbox pattern, not a broker miracle).

Ask whether the interaction is a queue at all:

request/response with a human waiting     → HTTP, not a queue (a 3 s queue = a 3 s page)
needs strict global ordering              → one partition/queue = one lane; think twice
needs read-your-writes consistency        → queue breaks it; use the primary
work that can lag by 30 s and retry       → queue — this is the sweet spot

Delivery semantics: the three and a half options

# at-most-once: send it, forget it. Duplicates impossible, losses possible.
producer.send(msg)                      # fire and forget — metrics, cache warmups
 
# at-least-once: ack only after success. Losses impossible, duplicates possible.
while True:
    msg = queue.receive()               # SQS visibility timeout / Kafka fetch
    process(msg)                        # may crash here
    queue.ack(msg)                      # unacked → redelivered later
 
# effectively-once: at-least-once + idempotent processing
# (the only kind that exists end to end)
def process(msg):
    with db.transaction():
        insert_into_processed(msg.id)   # primary key → duplicate rejected
        apply_effects(msg)

There is no at-least-once without duplicates and no exactly-once without idempotency. Crash between process and ack, and the message comes back; your handler runs twice. Kafka's producer idempotence (per-partition sequence numbers) and transactional offset commits narrow the duplicate window inside the broker, and SQS's visibility timeout does the equivalent — but effects (charges, emails, counters) are outside the broker's reach. The duplicate probability is not exotic: any consumer that crashes at a rate of one per few thousand messages will redeliver, and retries after deploy rollouts redeliver at a much higher rate. Build the dedup key first:

BEGIN;
INSERT INTO processed_messages (message_id, processed_at)
VALUES ('order-123', now());          -- primary key on message_id
-- duplicate → unique violation → ack and skip
UPDATE orders SET status='shipped' WHERE id = 123;
COMMIT;

Ordering: per-partition, and why that's the whole story

Kafka orders messages within a partition, never across them. Consumers in a group split partitions, so parallelism and ordering are in direct tension:

producer.send('orders', value=order, key='customer_42')
# key → partition by hash(customer_42): every event for this customer
# lands in one partition → consumed in order by exactly one consumer
 
# per-customer ordering, across customers in parallel — the usual goal
#
# global ordering requires 1 partition → 1 consumer → throughput capped
# at roughly tens of MB/s and a single core's worth of processing

Reordering happens in practice even when the log is perfect: two consumers handling different partitions write to a shared table, a retry lands after a newer event, or your handler processes with concurrency > 1. The rule that survives: make handlers idempotent and monotonic (apply state changes with compare-and-set on a version) rather than trusting the pipe to be unbreakable.

Kafka, RabbitMQ, SQS: different machines

              Kafka                    RabbitMQ               SQS
mechanism     distributed commit log    broker + exchanges     managed queue
ordering      per-partition             per-queue (FIFO mode)  standard: best-effort;
                                                                FIFO: per group
delivery      at-least-once;            at-least-once (ack)    at-least-once
              exactly-once with
              transactions + idempotent
              producer
retention     days–forever (replay!)    until consumed         up to 14 days
throughput    ~millions msg/s cluster   ~tens of thousands/s   ~tens of thousands/s
                (100k+/s per partition)
routing       key → partition           exchanges, bindings    topic filters
ops          run the cluster           run the cluster        zero ops

The load-bearing differences are retention and routing. Kafka's log is a replayable database: a new consumer can start at offset 0 from last Tuesday, and reprocessing after a bug fix is a rewind rather than a restore. RabbitMQ's exchanges give you fanout and pattern-based routing (order.*.shipped) with low per-message latency, at the price of queue-per-consumer growth and cluster operations. SQS gives you the durability and visibility-timeout model with no brokers to run — and long polling (WaitTimeSeconds=20) to avoid paying for empty receives.

sqs.receive_message(QueueUrl=q, MaxNumberOfMessages=10, WaitTimeSeconds=20)
# visibility timeout: message invisible while being processed (default 30 s)
# if processing takes longer → another consumer gets it too → duplicate
# set visibility > p99 processing time, or heartbeat/extend it

Backpressure, poison messages, and the depth alarm

A queue with no upper bound converts overload into memory pressure — you want a bounded queue and a deliberate rejection policy at the producer. On the consumer side, the poison message problem is structural: a message that always crashes the handler will be redelivered forever unless redelivery is capped:

# SQS redrive policy — cap redeliveries, then park the message
redrivePolicy:
  deadLetterTargetArn: arn:aws:sqs:…:orders-dlq
  maxReceiveCount: 5        # after 5 failed receives → DLQ, alarm fires
visibilityTimeoutSeconds: 60   # > p99 handler time (~18 s) with headroom

Monitor what the depth means: ApproximateAgeOfOldestMessage growing while depth is flat means one stuck consumer; depth growing uniformly means you are behind capacity. Each has a different fix, and neither is fixed by adding producers' retries.

Key takeaways

• Queues buy time, rate, and availability — not contracts or consistency. Use an outbox when the write and the event must agree.
• Exactly-once doesn't exist end to end. Design idempotent consumers (dedup table with a primary key) before you optimize delivery semantics.
• Ordering is per-partition: key by the ordering dimension, keep consumer concurrency within the partition count, and make handlers monotonic anyway.
• Choose by the replay/routing/ops trade: Kafka for a replayable log, RabbitMQ for routing topology, SQS when zero ops beats both.
• Bound everything: producer-side admission, visibilityTimeout > p99, maxReceiveCount into a DLQ with an alarm, and alert on oldest-message age.