SLIs, SLOs, and Error Budgets: Reliability as a Number

Reliability is not a mood. "The site feels faster lately" and "that outage was really bad" are both opinions; an error budget is an agreement. The vocabulary that makes the agreement precise comes from SRE practice: an SLI (service level indicator) is the measurement, an SLO (service level objective) is the target, and the error budget is the allowance you get when real life misses the target.

The SLI: choose what actually matters to users

An SLI is a ratio — good events divided by total events — measured over a window. The classic example is request availability:

SLI availability = successful_requests / total_requests

The trap is defining "successful". A 500 from the load balancer is an obvious failure. But a request that returns 200 after 30 seconds may be more damaging to users than a fast 503, so a good availability SLI for a user-facing API usually comes in two parts: availability (HTTP 5xx or no response) and latency (responses under a threshold like p99 ≤ 400 ms). Each is a separate SLI. Common choices besides request-based ones:

• Throughput-based: events successfully ingested / events offered to the pipeline.
• Freshness-based: the fraction of time the newest data committed is under N seconds old (a feed, a replica, a cache invalidation).
• Durability-based: the fraction of writes that survive a crash.

Notice what SLIs are not: they are not internals. CPU utilization, disk queue depth, and garbage-collection pauses matter only when they move a user-visible SLI. You can have a perfectly healthy cluster serving a terrible search experience, and an unhealthy cluster proudly meeting its SLI because the SLI measured the wrong thing.

The SLO: the number with a time window

An SLO is "99.9% of requests complete successfully, measured over the trailing 28 days" — the ratio and the window. The window is not decoration. It decides how much unreliability you tolerate per event and how fast you must react:

• Per-event tolerance. At 99.9% availability, a 10-minute outage against a 24/7 baseline burns about 23% of the ~43-minute monthly budget (0.1% of a 30-day month). At 99.99%, the same outage burns 231% — ten times more. The higher the target, the smaller every incident's allowance.
• Reaction time. A weekly window is appropriate for batch workloads and internal tooling; a rolling 28-day window is the SRE default for user-facing services. Some teams layer a short "burn alert" window (say, last 30 minutes) on top so a spike triggers a page long before the monthly budget is gone.

Choose the target from what the product needs, then cost it. Adding a 9 is not free: 99.9% might tolerate a single full timezone outage per month, while 99.99% leaves about 52 minutes of downtime per year — essentially demanding multi-region redundancy, retries, and queuing. A number you cannot operationally defend is not a goal, it is a prayer.

The error budget: engineering's allowance

The error budget is 100% − SLO. At a 99.9% SLO you have a 0.1% budget: about 43 minutes of "unreliability" per month, all incidents included. The radical move is to stop treating that budget as a failure and start treating it as a currency:

• When the budget is full, risk is cheap. You can ship the risky refactor, deploy the rewrite, rotate the dependency that is overdue for an upgrade. This is the product side of the contract: velocity is granted by reliability, not granted despite it.
• When the budget is depleted, the product stops releasing non-essential changes until reliability recovers. Feature work freezes; the engineering team works the incident backlog. Nobody argues about priorities anymore — the number decides.

Two numbers make the budget actionable. The error budget burn rate tells you how fast you are consuming the budget (burn rate = observed error rate / budget error rate; a burn rate of 14.4 sustained for 24 hours consumes about half of a 28-day budget of 0.1%, and Google SRE pairs a 14.4 × 1-hour burn with a 2% budget-consumption alert). The alert threshold pages you when consumption is fast, e.g. "budget 5% consumed in the last hour."

Where it bites in production

Three failures of this system are worth knowing before you adopt it:

1. Good SLI, wrong measurement window. Averaging over 28 days hides an afternoon of pain. Always pair the long window with a short burn-rate alert.
2. Targets that lie about dependencies. If your availability is 99.99% but your biggest dependency advertises 99.9%, your users experience roughly 99.9%. Your SLO must account for the dependency chain, or you will fail on days you did nothing wrong.
3. Latency SLIs averaged instead of percentile. An average latency of 120 ms is consistent with 10% of users seeing 3 seconds. Use p50/p95/p99 and keep the threshold in the SLI definition.

Key takeaways

• Choose SLIs from what users experience (availability + latency per request), not from cluster internals.
• Set SLOs with an explicit window; the window decides how costly each incident is.
• The error budget turns reliability from a subjective argument into an operational contract that gates releases.
• Alert on error-budget burn rate, not on raw error rate, so you page on the thing that threatens the budget.

You have a working agreement the day you can answer: what is the SLI, what is the SLO, what burned last week, and what slowed down because of it?