Reliability Comes From Explicit Failure Behavior

An unreliable component does not make a system unreliable if the system contains and recovers from its failure. Reliability is therefore not a property of parts — every part fails eventually — but of the behavior written around each failure: what the caller sees, how far it spreads, how detection works, and how service returns.

Name the failure modes first

Start with a table, not a slogan. For each dependency, write the mode, the user-visible behavior, the detection signal, and the recovery:

dependency        failure mode      user sees            detected by        recovers by
----------------  ---------------   ------------------   ----------------   ----------------
payment provider  timeout (2 s+)    checkout blocked     breaker + p99      provider recovers;
                                                                  probe   fallback PSP
recommendations   5xx / slow        home page without    error-rate alert  cache fallback
                                                tiles
cache (Redis)     connection loss   slower reads, then   pool errors +     reconnect with
                                     fallback to DB      db latency rise   backoff; cache warms
primary database  failover (30 s)   write errors,        replication lag   automatic promote;
                                     reads from replica                  clients retry
deploys           bad config        elevated errors      canary halt       rollback (flags)

The act of filling the table is the design work: it forces you to distinguish blocking from degrading per dependency, which is the decision that determines whether an outage in one service becomes an outage in the product.

For the check question — a service depending on a payment provider and a recommendation service: the payment outage should block checkout (charging is the transaction; guessing is worse than failing, and a declined payment costs more than an abandoned cart). The recommendation outage should degrade — drop the rail, render the page from cache, keep the core path. Two dependencies, same "unavailable," opposite answers.

Timeout, breaker, bulkhead: the containment trio

Timeouts bound waiting. Derive them from your own latency budget, not the dependency's documentation: a 200 ms p99 budget with 60 ms of local work leaves at most ~120 ms for a remote call — set the timeout there and let the caller fail into its fallback. A default 30-second timeout against that budget converts a slow dependency into a thread-pool exhaustion in one bad minute.

Bulkheads bound blast radius. Separate connection pools or concurrency caps per dependency so that a stalled provider cannot occupy every worker. The failure mode you are preventing is the classic one: 100% of threads waiting on one dependency, so requests that never touch it also hang.

Circuit breakers bound repeated attempts. Stop calling a failing dependency, let it recover, then probe carefully:

closed    : requests flow; failures counted in a rolling window
            open when failure rate > 50% over 10 s AND ≥ 20 calls in window
open      : fast-fail immediately (error in <1 ms instead of a 120 ms timeout)
half-open : after 15 s, admit 3 probe requests
            all succeed → closed; any fails → open again for 15 s

The load math is why this matters. Ten callers each calling a dead dependency with a 120 ms timeout and 3 retries generate up to 4× the normal request count against a service that is already failing (1 attempt + up to 3 retries each) — retry storms are self-inflicted second outages. Cap retries (2 attempts plus jitter), and make sure retries land only on idempotent operations.

Redundancy that isn't independent is decoration

Two replicas behind one load balancer fail together if they share the failure mode: the same AZ power feed, the same bad config shipped in the same artifact, the same shared database downstream, the same expired certificate. Independence is what redundancy buys, and it has to be designed — different zones, staged rollouts so one cohort is never 100% of capacity, and circuit breakers on shared dependencies.

And untested failover is a hypothesis. If the promotion path has not run in the last 90 days, assume it will fail when you need it: the runbook rots, the credentials expire, and the DNS TTL turns out to be six hours. Fire drills (planned primary shutdown during a quiet hour) are the cheapest reliability investment available.

Availability needs a definition and a window

"99.9% available" means nothing until you fix two things: the window and what counts as a successful request.

99.9%  = 43.8 min of full outage per month
99.99% = 4.4 min per month          (10× stricter)
p99 latency counts as unavailability?  95%? — decide in advance
partial degradation (recommendations down) — counted or not?
window: calendar month (not rolling) — simpler to communicate

Include latency in the SLO or you will ship an "available" service that takes 8 seconds per page. And decide how partial failures count before the incident, not during the argument.

Key takeaways

• Reliability is contained failure plus tested recovery. Write the table — mode, user visible behavior, detection, recovery — for every dependency; the blocking-versus-degrading call is the design.
• Timeouts from your budget, bulkheads per dependency, breakers with a probe. Together they keep one failure from becoming thread exhaustion everywhere else.
• Retries are load multiplication against a sick dependency: cap them, jitter them, and keep them off non-idempotent operations.
• Redundancy without independence and drills is decoration — shared AZs, shared configs, and untested failovers all fail together.
• Define the window and the success signal for every availability number you publish, including how partial degradation counts.

Google SRE Book covers error budgets, SLOs, and alert design in depth.