Reliability is not a feeling — it is a measured ratio compared against a target, with a budget for how much you are allowed to fail. Everything in this chapter hangs off four nested ideas: you measure a signal (SLI), you target it internally (SLO), you promise a looser version externally (SLA), and the gap below perfection is your error budget — the fuel you spend on shipping.

SLI vs SLO vs SLA vs Error Budget

These four terms get muddled in interviews. Keep them crisp: an SLI is what you measure, an SLO is what you aim for, an SLA is what you contractually promise, and the error budget is the slack between your SLO and 100% that you are permitted to burn.

Error‑budget math (the number to memorise)

Pick a window (28 or 30 days is common). The budget is the complement of the SLO applied to that window:

# 99.9% monthly SLO, 30-day window
budget_fraction = 1 - 0.999      # = 0.001  (0.1%)
month_minutes   = 30 * 24 * 60   # = 43,200 min
budget_minutes  = 0.001 * 43200  # ≈ 43.2 min/month of allowed badness

Burn‑rate alerting: don’t alert on the raw SLI

Alerting the instant the SLI dips is noisy. Instead alert on burn rate — how fast you are consuming the budget relative to the window. A burn rate of 1× exhausts the whole month’s budget in exactly a month; 14.4× burns it in ~50 hours. Use multi‑window, multi‑burn‑rate alerts: a fast‑burn window catches acute outages quickly, a slow‑burn window catches quiet chronic erosion without paging at 3 a.m. for a blip.

Alert class	Budget consumed	Long / short window	Burn rate	Action
Fast‑burn	2% of monthly budget	1 h & 5 min	14.4×	Page on‑call now
Medium‑burn	5% of monthly budget	6 h & 30 min	6×	Page (business hours OK)
Slow‑burn	10% of monthly budget	3 d & 6 h	1×	Open a ticket

The short window is a reset guard: both the long and short window must be burning for the alert to fire, so it clears quickly once the incident is mitigated instead of ringing for another hour.

The availability “nines”

“How many nines?” is shorthand for how much downtime you tolerate. Memorise the top rows — three and four nines are what most interview systems target.

Availability	Nickname	Downtime / year	Downtime / month	Downtime / day
90%	one nine	36.5 days	72 hours	2.4 hours
99%	two nines	3.65 days	7.2 hours	14.4 min
99.9%	three nines	8.76 hours	43.2 min	1.44 min
99.95%		4.38 hours	21.6 min	43.2 s
99.99%	four nines	52.6 min	4.32 min	8.64 s
99.999%	five nines	5.26 min	25.9 s	0.86 s

Availability Zones & Regions — where the redundancy lives

Those nines don’t come from one very reliable machine — they come from redundancy across independent failure domains. Cloud providers hand you a ready‑made hierarchy of them: a Region contains several Availability Zones, and you place copies of your system across a few so that losing one doesn’t take you down.

Rule of thumb: spread every stateless tier and your database across ≥ 2–3 AZs in one region for everyday high availability — a whole AZ can drop and you just lose a slice of capacity, not the service. Add multi‑region only when you need disaster recovery or global latency, because it’s the expensive, complicated tier.

Latency percentiles — why the mean lies

Latency distributions are right‑skewed: most requests are fast, but a long tail of slow ones drags the average to the right of the typical experience. A mean of 55 ms can hide a p99 of 250 ms. Report percentiles, not averages: pN is the value below which N% of requests fall.

Fan‑out amplifies the tail

In a microservice request that fans out to many leaves, the request is only as fast as its slowest dependency. If one request touches 100 services and each has a p99 of 250 ms, the chance that none of them is in its slow tail is tiny — so the tail of a leaf becomes the common case for the whole request.

Timeouts & deadline propagation

An unbounded wait is a resource leak waiting to become an outage: threads, connections and memory pile up behind one stuck dependency until the whole service falls over. Every remote call needs a timeout. Distinguish the kinds:

Timeout	What it bounds	Typical value
Connect	Establishing the TCP/TLS connection	50–200 ms
Read / socket	Gap between bytes once connected	100 ms – 2 s
Total / deadline	End‑to‑end wall clock for the whole operation	the request’s budget

The powerful idea is a deadline (budget) propagated across hops. The client sets one absolute deadline; every downstream call gets whatever time remains, minus a small safety margin. A hop that sees the budget already exhausted fails fast instead of doing doomed work.

// One absolute deadline, computed once at the edge
deadline = now() + total_budget        // e.g. now + 1000ms

function call(downstream):
  remaining = deadline - now()
  if remaining <= safety_margin:
    throw DeadlineExceeded   // don't start work we can't finish
  downstream.timeout = remaining - safety_margin
  return downstream.send()

Retries, exponential backoff & jitter

Retries paper over transient faults (a dropped packet, a brief 503, a leader election). But retry the wrong thing and you amplify an outage. Two rules gate every retry:

The backoff formula

Grow the delay geometrically and cap it: delay = min(cap, base · 2^attempt). With base = 100ms and cap = 6.4s the delays are 100, 200, 400, 800, 1600, 3200, 6400 ms… This spaces out retries so a struggling backend gets breathing room instead of a wall of duplicates.

Why jitter is non‑negotiable

Without jitter, every client that failed at the same instant retries at the same instant — a synchronised retry storm (thundering herd) that hits the recovering service with a perfectly aligned spike and knocks it back down. Adding randomness spreads the retries out. Two standard schemes (from the AWS “backoff and jitter” playbook):

temp = min(cap, base * 2**attempt)

# Full jitter — pick anywhere in [0, temp]
sleep = random_between(0, temp)

# Equal jitter — keep half, randomise the other half
sleep = temp/2 + random_between(0, temp/2)

# Decorrelated jitter — walk based on the previous sleep
sleep = min(cap, random_between(base, prev_sleep * 3))

Circuit breakers (reliability lens)

A circuit breaker stops you from hammering a dependency that is already down, converting slow cascading failures into fast, cheap rejections that protect the error budget. It’s a small state machine wrapped around a call. (The full mechanics live in Chapter 1; here’s the reliability‑relevant recap.)

Idempotency (reliability lens)

Networks give you at‑least‑once delivery by default: a client can’t tell a lost request from a lost response, so it retries — and the server may process the same operation twice. The cure is making operations idempotent: repeating them has the same effect as doing them once.

Incident handling

When the budget is burning, process beats heroics. Classify severity, mitigate first, and learn without blame.

Severity	Impact	Response
SEV1	Major outage / data loss / core flow down for many users	Page immediately, all‑hands, incident commander, comms to customers
SEV2	Significant degradation, a region or feature down, SLO at risk	Page on‑call, dedicated responder, status update
SEV3	Minor / contained, workaround exists, budget mostly intact	Ticket, handle in business hours

The lifecycle

1. Detect. Alerting (burn‑rate SLO alerts, health checks, synthetics) surfaces the problem. The clock on MTTD (Mean Time To Detect) starts here.
2. Triage & acknowledge. On‑call ack’s the page — that stops MTTA (Mean Time To Acknowledge) — sets a severity, and — for SEV1/2 — names an Incident Commander who coordinates while others investigate.
3. Mitigate. Stop the bleeding before understanding it: roll back the deploy, fail over, shed load, disable the bad feature flag. Restoring the SLO is the goal, not the root cause.
4. Resolve. Confirm the SLI is healthy and the budget has stopped burning. MTTR (Mean Time To Recovery/Repair) stops here.
5. Learn. Write a blameless postmortem: timeline, contributing factors (systems, not people), and concrete action items with owners. Feed fixes back into runbooks and alerting.

Memorise this