Choose Tests by the Failure They Can Catch

Tests provide evidence about behavior at different boundaries. The useful framing is not "how much coverage do we have" but which production failure would this test have caught? Each level trades feedback speed, realism, and setup cost, and the strategy is choosing the level where the failure is visible — and no higher than necessary.

What each boundary can actually prove

layer         real pieces in play           catches                  cannot catch
------------  ---------------------------   -----------------------  ------------------
unit          pure logic, fakes             decision rules,         wiring, schema,
                                            edge cases, budgets     driver behavior
integration   real DB/queue/broker,         persistence semantics,  full user journey,
              fake external services        transactions, retries,  cross-service flows
contract      both sides via schema         field renames,          runtime behavior
              definitions                   incompatible changes
end-to-end    deployed app + real deps      journeys, config,       deep edge cases
                                            migrations, routing     (too slow to enumerate)

The three concrete examples from the model become precise at their own level:

• Unit: the retry policy stops after its configured budget — assert attempts == 3 and the backoff schedule with a fake clock. No I/O needed; runs in microseconds; the cheapest place to enumerate every branch (timeout, 429, 4xx, connection reset).
• Integration: the persistence adapter records the idempotency key and the effect in one transaction — this requires a real database, because the property under test is transactional, and a mock cannot fail to roll back.
• End-to-end: checkout through the deployed path returns the expected receipt — this is the only level that exercises config, migrations, routing, and auth together.

The check question: mocked database, lost updates in production

A service test passes with a mocked database, but production loses updates under concurrent requests. Two things are missing: the layer (integration against the real database) and the invariant (read-modify-write must be atomic):

-- the defect, in SQL terms: check-then-act without a lock
BEGIN;
SELECT qty FROM inventory WHERE sku='A1';        -- reads 10 (so does txn 2)
UPDATE inventory SET qty = qty - 1 WHERE sku='A1';  -- both write 9
COMMIT;                                          -- one decrement vanished
 
-- the invariant a new test must exercise: two concurrent transactions
-- that each decrement must leave qty = initial − 2
BEGIN;
SELECT qty FROM inventory WHERE sku='A1' FOR UPDATE;  -- second txn blocks
UPDATE inventory SET qty = qty - 1 WHERE sku='A1';
COMMIT;

The test: run two concurrent decrements (threads, or two sessions with a barrier), assert the final value equals initial − 2, and repeat 100 times — flaky-loss bugs surface as a probabilistic failure, which is why concurrency tests need repetition and a seed. The mock never failed because the mock had no locking semantics to get wrong.

Feedback latency is the real budget

A suite's value decays with its wall-clock time. Teams route around slow suites — they batch changes, skip the slow jobs, or approve red — and the fastest way to lose test discipline is to let CI drift past ~10 minutes:

static checks + lint              ~10 s
unit (thousands of tests)         ~90 s
integration (real Postgres/Redis) ~2 min
contract (3 dependencies)         ~20 s
smoke E2E (critical path only)    ~90 s
                                 ────────
                                 < 6 min PR-to-green

Slow, low-value tests belong in nightly, not in every PR — moving them is better than deleting them, as long as somebody owns the nightly triage.

Costs and failure cases

• Coverage is not assertion. 90% line coverage with assertions that only check "no exception thrown" proves almost nothing. The metric that matters is which behaviors are pinned: assert outcomes and invariants, not internal call sequences.
• Tests that mirror implementation block refactoring. If a test asserts that method X called helper Y twice, renaming a private helper fails the suite with zero behavioral change. Test observable behavior; a suite that fails on harmless refactors will be worked around rather than fixed.
• Flaky tests are worse than missing tests. A test that fails 1% of the time trains everyone to re-run CI — at which point every test is optional. Quarantine and repair flaky tests immediately; a rerun button is a slowly spreading outage of your safety net.
• A few E2E tests cover little. Two happy-path journeys say nothing about failure modes; prefer pushing failure cases down to integration tests where they are faster and more deterministic, and keep E2E for the journeys that must never break.

Key takeaways

• Pick the level where the failure is visible. Transactional properties need a real database; routing needs a deployed app; a retry budget needs only a fake clock.
• Write the test against the invariant, not the implementation: final = initial − 2 under concurrency, attempts ≤ 3 under timeout, stored response replayed on key reuse.
• Guard the feedback budget (< ~6 minutes PR-to-green) and treat flakiness as a defect, because a flaky suite is a disabled suite with extra steps.
• Coverage measures code touched; assertions measure behavior pinned. Only one of them survives a production incident.

The Practical Test Pyramid (martinfowler.com) argues for exactly this balance of layers.

Production cost model and latency numbers

To apply Choose Tests by the Failure They Can Catch effectively in production systems, engineers must move beyond qualitative diagrams and quantify the exact hardware and network costs at each boundary:

Failure modes, debugging, and operational checklist

In real incidents, systems rarely fail at average (p50) load—they fail when a slow dependency, skewed hot key, or resource leak pushes concurrency across a non-linear saturation cliff. When reviewing code or architecture that relies on Choose Tests by the Failure They Can Catch, walk through three verification questions:

1. What is the invariant under partial failure or contention? If a worker crashes mid-operation, a network packet is delayed and reordered, or two threads race on the same state transition, prove which atomic primitive, version number, or idempotency key prevents silent corruption.
2. How is backpressure enforced when demand exceeds capacity? Unbounded in-memory queues, unthrottled retries, and blocking calls without explicit deadlines turn a localized 100 ms slowdown into a fleet-wide Out-Of-Memory (OOM) cascade. Always bound queue depths, propagate caller deadlines, and shed excess load early.
3. Which metrics and traces isolate the bottleneck in under 60 seconds? Instrument saturation, queue wait time vs service execution time, cache hit/miss ratios, and error rates by failure reason so on-call engineers can immediately distinguish a compute bottleneck from an I/O or lock-contention stall.