Your primary database is your best source of truth, but the rest of your system does not live there: the search index, the cache, the analytics warehouse, the notification pipeline. The classic way to keep all those copies consistent was to sprinkle write-path logic through every service that wrote data — dual-writes. Every one of those writes is a chance for a partial failure where the database committed and the index did not. Change data capture (CDC) inverts the problem: you read the database's own record of changes and let the copies follow.
What a CDC capture actually reads
Databases already have an ordered, durable stream of every change: the write-ahead log (PostgreSQL's WAL, MySQL's binary log, MongoDB's oplog). A CDC connector positions itself as a replica of that log and replays it to a destination — a message broker like Kafka, another store, or a search engine. The connector approach matters for three reasons:
- It is complete. Every committed change appears, including changes made by SQL that no application code touched (
UPDATE ... SETin a migration, a manual fix in prod). - It is non-invasive. No application changes, no trigger logic, no bespoke dual-writes.
- It is ordered per key. The log gives you the order of commits, which is the ordering your downstream systems usually need.
The capture mechanism in modern connectors (Debezium, the default in most stacks) is log-based, with one tool maintaining its own checkpoint of the log position. On first run it takes a snapshot of existing rows, then switches to streaming changes from the snapshot's position. The row image for each change carries metadata: before, after, op (c / u / d / r for create, update, delete, read), and the transaction id.
The output schema: an envelope with intent
A streamed change record is more than the row — it is an intent a consumer can reason about:
{
"op": "u",
"before": {"id": 42, "price": 1999},
"after": {"id": 42, "price": 2499},
"ts_ms": 1728...
}Four properties make this the workhorse:
- The event is materialized as data. Consumers can compute consequences (re-index the product, invalidate the cache, emit the price-change notification) without re-querying the source.
- Out-of-order is rare and detected. If consumers need exactly-once semantics, the transaction id lets you dedupe and the LSN lets you checkpoint.
- Deletes are visible. A cache or index that only sees inserts and updates leaks stale rows;
op: "d"fixes the leak. - It is replayable. The log is the raw material for rebuilding the search index from scratch ("bootstrap"), testing consumer logic on real change history, and auditing "what actually changed".
The failure modes that bite
-
At-least-once delivery, not exactly-once. A consumer that crashes after processing but before committing its offset will reprocess the record. The standard defense is idempotent consumers: keyed by a primary key / event id, upserting instead of inserting, so reprocessing is a no-op.
-
Lag as a budget. CDC is asynchronous, so your cache and index are slightly stale by design. The number every pipeline needs is replication lag — the age of the newest committed change against the freshest processed change. Metrics like "newest row seen by search" are the SLI for the whole downstream.
-
Schema evolution. A column added or renamed breaks consumers that assume shape. Versioned event schemas (Avro + Schema Registry, or
PROTOenvelope withschema_version) let old and new consumers coexist during the migration window. -
The speed-of-light problem is not solved. CDC is great for keeping a search index within seconds of primary. It is not a substitute for the transactional guarantee you get from reading the primary within the same transaction — analytics and caches tolerate lag; a balance read cannot.
The pipeline that pays for itself
The arrangement that has become standard in shipping systems:
Postgres ──► Debezium ──► Kafka(topic: db.public.orders) ──► {search index, cache, warehouse, notifications}One connector, one ordered stream, N consumers each reading the envelope and materializing its own view. Every consumer is replayable from the log, and none of them touches the write path. The moment you find yourself maintaining N write-side "also update the index" calls, you have already paid the dual-write tax — CDC is the cheaper contract, and it is the one that keeps copies consistent with the source that actually matters.
Key takeaways
- CDC treats the database's write-ahead log as the single source of change events — complete, ordered per key, and non-invasive.
- The change envelope (op, before, after, transaction metadata) is materialized data consumers can reason about and replay.
- Handle at-least-once with idempotent consumers; monitor replication lag as your pipeline's SLI.
- Use CDC to feed caches, search, analytics, and notifications — not to replace transactional reads of the primary.
Production cost model and latency numbers
To apply Change Data Capture: Databases as the Source of Truth for Everything Else effectively in production systems, engineers must move beyond qualitative diagrams and quantify the exact hardware and network costs at each boundary:
| Operation / Boundary | Typical Latency | Dominant Bottleneck | Production Mitigation |
|---|---|---|---|
| L1 / L2 Cache Access | 1 ns – 4 ns | Register dependency chains & pointer chasing | Contiguous memory layouts, cache-line alignment (64 B), and batch iteration |
| Main Memory (DRAM) Fetch | 60 ns – 100 ns | TLB misses, random heap indirection, NUMA hops | HugePages (2 MB), arena/slab allocators, and pre-allocated ring buffers |
| Kernel Syscall / Context Switch | 1 µs – 5 µs | User-to-kernel mode switch, page-table flush, lock contention | Non-blocking readiness loops (epoll / io_uring), batching, and zero-copy I/O |
| NVMe SSD Random Read / fsync | 50 µs – 500 µs | Block I/O queue depth, write amplification, WAL flushes | Group commit, sequential append logs, and bounded buffer-pool pinning |
| Intra-Region RPC Roundtrip | 0.5 ms – 2.5 ms | Serialization, TCP/TLS framing, tail queueing ($L = \lambda W$) | Connection pooling, deadline propagation, and p95 hedged requests |
Failure modes, debugging, and operational checklist
In real incidents, systems rarely fail at average (p50) load—they fail when a slow dependency, skewed hot key, or resource leak pushes concurrency across a non-linear saturation cliff. When reviewing code or architecture that relies on Change Data Capture: Databases as the Source of Truth for Everything Else, walk through three verification questions:
- What is the invariant under partial failure or contention? If a worker crashes mid-operation, a network packet is delayed and reordered, or two threads race on the same state transition, prove which atomic primitive, version number, or idempotency key prevents silent corruption.
- How is backpressure enforced when demand exceeds capacity? Unbounded in-memory queues, unthrottled retries, and blocking calls without explicit deadlines turn a localized
100 msslowdown into a fleet-wide Out-Of-Memory (OOM) cascade. Always bound queue depths, propagate caller deadlines, and shed excess load early. - Which metrics and traces isolate the bottleneck in under 60 seconds? Instrument saturation, queue wait time vs service execution time, cache hit/miss ratios, and error rates by failure reason so on-call engineers can immediately distinguish a compute bottleneck from an I/O or lock-contention stall.