Event-Driven Architecture: When It Actually Pays

Event-driven architecture (EDA) replaces the request-response conversation with a different primitive: a producer publishes a fact, and nobody in the producer's process knows who cares. The event is durable data; consumers subscribe, read, and act on their own schedule. That inversion buys real capabilities — fan-out, tolerance to slowness, independent evolution — but it also hands you the hardest problems in distributed systems: duplicate delivery, ordering, and multi-step failure. This article is about the mechanics of doing it correctly, and the exact conditions where the trade is worth making.

When events beat RPC

A synchronous call is the right shape when the caller needs the answer to continue: "authenticate me", "reserve a seat". Events win when at least one of these holds:

1. Multiple independent consumers need the same fact. "Order placed" drives inventory, billing, analytics, email, and search indexing. Five synchronous calls make the original request latency and failure surface a function of five downstream systems. One event makes it a function of one publish.
2. The consumer can tolerate delay. Analytics, search indexes, and email do not need the data within the user's request round-trip. Given delay tolerance, the async path costs nothing in user experience.
3. You can absorb partial failure. The moment a downstream system is allowed to be down for a while and catch up later, the whole system's availability decouples from the slowest component.

If none of those hold — the consumer must answer synchronously, the data must be immediately consistent, the flow is a single request → single response — events are the wrong tool, and RPC is simpler.

The dual-write problem and the outbox pattern

The hard part of EDA is not publishing; it's publishing exactly once per state change when the state lives in a database. If you write the row and then publish the event, you have two writes with no transaction between them: the publish can fail after the commit, and the event is lost forever.

The outbox pattern solves this with one local transaction:

BEGIN;
INSERT INTO orders (id, status) VALUES ('o-1', 'PAID');
INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at)
VALUES ('e-1', 'o-1', 'order.paid', '{"orderId":"o-1"}', now());
COMMIT;

A relay process — either a poller or a transactional log tailer — reads undelivered outbox rows and publishes them to the broker:

while True:
    for event in fetch_undelivered(limit=100):
        broker.publish(event.topic, event.payload)
        mark_delivered(event.id)

The event and the state change are now atomic in the only place that can guarantee it: the database. The relay is crash-safe because an event is only marked delivered after the publish succeeds — and if the publish succeeds but the mark fails, the event is published twice. That is not a bug; it is the design. See what that means next.

At-least-once delivery is the contract

Brokers deliver at-least-once: every message arrives one or more times, in order per partition but without global guarantees. The consumer's code, not the broker, must make duplicates harmless:

def on_order_paid(event):
    if not inventory.has_reserved(event.order_id):
        inventory.reserve(event.order_id)   # idempotent by key

Exactly-once delivery is a lie (see our article on delivery semantics); at-least-once plus idempotent consumers is the only honest contract. Every handler must be written against the possibility that it already ran. The outbox's duplicate-on-crash behavior is why.

Sagas: the price of no global transaction

When one fact triggers changes in several services, you cannot wrap them in a database transaction. A saga is the alternative: a sequence of local transactions with compensating actions. If step three fails, you run the undo of step two, then step one.

The tradeoffs are structural, not cosmetic:

• Intermediate states are visible. Between steps, the system is partially updated — an order can be paid in billing but not reserved in inventory. Consumers must tolerate these states.
• Compensations are not transactions. The compensation for "payment charged" can itself fail, and then the system is permanently inconsistent until a human or a reconciliation job intervenes.
• The saga is your code, not your database. Every failure path you previously got from ACID — rollback, isolation, atomicity — you now implement by hand.

A saga is worth it only when the alternative — a single service owning the whole flow — is genuinely worse. If one service can own the flow, the saga is self-inflicted complexity.

The honest tally

Events beat RPC under three conditions: multiple consumers, delay tolerance, and absorbable failure. They cost you: duplicate handling in every consumer, ordering reasoning per topic, a saga for every multi-service flow, and a relay you must operate. The systems that regret EDA are the ones that adopted it for the diagram, then discovered the duplicate in the billing handler. Adopt it for the measured conditions — and make the outbox the first thing you build.