Map of this chapter: we move from how a request is served (stateless compute + caching + the database tier + load balancing) to how the system protects itself (retries, backpressure, idempotency, circuit breakers) and finally how it fails gracefully (degradation). Each block includes the theory, a concrete example, and a diagram you can redraw on a whiteboard.

Stateless Service Design

A service is stateless when any instance can handle any request because it keeps no per‑client data in local memory between requests. All durable or shared state lives outside the process.

Where does the state actually live?

“Stateless” never means state disappears — it means you externalize it to a purpose‑built store:

State	Externalized to	Example
Session / identity	Signed client token	JWT or opaque cookie the client resends each call
Hot shared data	Distributed cache	Redis / Memcached for carts, counters, rate buckets
Durable records	Database	Postgres, DynamoDB — the system of record
Large blobs	Object store	S3 / GCS for uploads, exports, model artifacts
In‑flight work	Message queue	Kafka / SQS holds jobs so a crash doesn't lose them

Sticky sessions are an anti‑pattern

Sticky sessions pin a client to one instance so its in‑memory session keeps working. It looks convenient and quietly re‑introduces every problem statelessness solved.

Caching

Caching trades a little staleness and memory for a large drop in latency and load. The skill is choosing the right pattern, the right eviction/TTL, and a plan for the two hard parts: invalidation and stampedes.

Caching patterns — when to use which

Pattern	How it works	Best for	Watch out
Cache‑aside (lazy)	App checks cache; on miss loads DB and populates cache.	Read‑heavy, general default.	First hit is slow; risk of stampede on hot keys.
Read‑through	Cache library loads from DB on miss transparently.	Uniform read path, less app glue.	Cache becomes a dependency of every read.
Write‑through	Write goes to cache and DB synchronously.	Read‑after‑write consistency needed.	Adds write latency; caches data you may never read.
Write‑behind (write‑back)	Write to cache, flush to DB asynchronously.	Write‑heavy, absorb bursts.	Data loss window if cache dies before flush.
Refresh‑ahead	Proactively refresh popular keys before they expire.	Predictable hot keys, latency‑critical.	Wasted work if prediction is wrong.

TTL & eviction

TTL (time‑to‑live) bounds staleness independent of memory pressure; eviction bounds memory independent of age. Use both: a TTL to expire, a policy to reclaim under pressure.

Invalidation — the hardest problem

“There are only two hard things in computer science: cache invalidation and naming things.” Strategies, roughly increasing in freshness and cost:

• TTL expiry — simplest; accept up to TTL seconds of staleness.
• Write‑invalidate — delete/overwrite the key on every write (shown above).
• Versioned keys — bake a version or hash into the key (user:42:v7); bump on change, old key ages out.
• Event‑driven — publish change events; subscribers purge affected keys (and CDN edges).

Cache stampede / thundering herd

A hot key expires and thousands of concurrent misses hammer the database at once. Mitigations, stackable:

Layered caching

Each layer absorbs load so the next one sees far less. The closer to the user the hit, the cheaper and faster it is.

Hit‑ratio latency math

Average latency is a weighted blend of hits and misses. With a 95% hit ratio, hits at 1 ms and misses at 50 ms:

# avg = hit_ratio·hit_latency + miss_ratio·miss_latency
avg = 0.95 × 1ms + 0.05 × 50ms
    = 0.95ms + 2.5ms
    = 3.45 ms

Databases & replication

The cache makes reads fast, but the database is the source of truth — the durable state everything else is derived from. Two questions dominate every data‑tier design: how does it survive a node dying (replication), and how does it hold more than one machine can (partitioning)?

Primary & replicas (leader–follower)

The default topology: one primary (a.k.a. leader) takes all writes; one or more replicas (followers) copy its changes and serve reads. Because most workloads are read‑heavy, fanning reads out to replicas is the cheapest way to add capacity — and if the primary dies, a replica can be promoted to take its place.

Synchronous vs asynchronous replication

Mode	How it works	On primary failure	Cost
Synchronous	Primary waits for the replica to confirm before acking the write.	No data loss — the standby is fully up to date.	Higher write latency; a slow replica stalls writes.
Asynchronous	Primary acks immediately; replicas catch up a moment later.	May lose the last few seconds of writes.	Fast writes, but replicas lag behind.

A common production setup blends both: one synchronous standby for zero‑data‑loss failover, plus several asynchronous read replicas for scaling reads.

Partitioning & sharding — scaling writes

Replicas scale reads, not writes: every replica still stores the whole dataset and replays every write. When a single primary can’t keep up with write throughput — or the data won’t fit on one machine — you shard: split the rows across N independent primaries, each owning a slice. The shard key decides which slice a row lands in, so pick one that spreads load evenly.

Strategy	How	Watch out for
Hash	shard = hash(key) % N.	Even spread, but range scans hit every shard; naive % N reshuffles everything when you add shards — use consistent hashing.
Range	Contiguous key ranges per shard (A–F, G–M…).	Range queries stay on one shard, but sequential keys (timestamps, auto‑increment ids) create hot shards.
Directory	A lookup table maps key → shard.	Most flexible and easy to rebalance, but the lookup service is a new dependency and potential single point of failure.

SQL vs NoSQL

Not a rivalry — two toolboxes with different defaults. Start relational unless a specific access pattern or scale requirement pushes you off it.

(Both columns are “good” — the winner depends on your access pattern, not on one being better.)

Reach for…	When
SQL (relational)	Relationships & joins, multi‑row transactions (money, orders, inventory), reporting/ad‑hoc queries, moderate scale. The safe default.
Document (MongoDB)	Self‑contained JSON documents, flexible/evolving schema, one aggregate fetched per read.
Key‑value (Redis, DynamoDB)	Simple get/put by key at massive scale — sessions, carts, feature flags, caches.
Wide‑column (Cassandra)	Huge write volume & time‑series/event data, known query patterns, linear horizontal scale.
Graph (Neo4j)	Deeply connected data where the relationships are the query — social graphs, fraud, recommendations.

Good practices

Load Balancing

A load balancer spreads traffic across the fleet and steers it away from unhealthy nodes. First decision: which OSI layer.

(Both columns are “good” here — the point is capability, not a winner. Pick L7 when you need content‑aware routing; L4 when you need raw speed.)

Algorithms

Algorithm	Picks backend by…	Use when
Round‑robin	Next in rotation.	Uniform requests & uniform backends.
Weighted RR	Rotation biased by capacity weight.	Heterogeneous instance sizes.
Least‑connections	Fewest active connections.	Long‑lived or variable‑length requests.
Least‑response‑time / EWMA	Lowest smoothed latency.	Backends with uneven or drifting latency.
Consistent hashing	Hash(key) mapped on a ring.	Cache/shard affinity; minimize reshuffle on scaling.

Why consistent hashing? With plain hash(key) % N, changing N remaps almost every key. On a hash ring, adding or removing a node only moves the keys in that node's arc — about 1/N of them — so caches and shards stay mostly warm during scale events. Virtual nodes smooth out the distribution.

Health & lifecycle

Global vs regional

Real systems layer them: global DNS/anycast/GeoDNS sends a user to the nearest healthy region; a regional L7 balancer then spreads within it.

Retries

Retries paper over transient blips — a dropped packet, a brief failover. They also amplify a struggling system into a full outage if done naively.

Backpressure & Load Shedding

You cannot process more than your slowest resource allows. Backpressure signals “slow down” upstream; load shedding drops work you can't serve so the work you do serve stays fast. Failing fast beats failing slow.

Little's Law — sizing the queue

The relationship between concurrency, arrival rate and latency:

// L = concurrent items in system,  λ = arrival rate,  W = time in system
L = λ × W

If requests arrive at λ = 500/s and each spends W = 0.2 s in the system, then L = 100 in flight. If your pool only handles 100 concurrently, a queue longer than that only adds latency without adding throughput — so bound the queue near L and shed the rest. Rearranged, W = L / λ: a deeper queue directly inflates response time.

Idempotency

An operation is idempotent if performing it once or many times yields the same result and side effects. It's what makes retries and at‑least‑once delivery safe.

Verb	Idempotent?	Why
GET	yes	Read‑only, no side effects.
PUT	yes	Replaces the resource to a fixed state; repeats converge.
DELETE	yes	Resource ends deleted regardless of repeat count.
PATCH	depends	Idempotent if the patch is absolute (set x=5), not if relative (x += 1).
POST	no	Creates/append by default — needs an idempotency key to be safe.

Idempotency keys + dedup store

Make a non‑idempotent write safe: the client sends a unique Idempotency-Key; the server records it with the result and returns the stored response on any replay.

POST /v1/payments
Idempotency-Key: 4b1e9f6a-2c...   # client-generated, unique per intent

# server logic
if store.exists(key):
    return store.get(key).response   # replay → same result, no double charge
result = charge_card(...)
store.put(key, result, ttl=24h)      # first time → do work, remember it

Circuit Breakers

A circuit breaker stops calling a dependency that's clearly failing, so you fail fast instead of piling requests onto a sick service (and stacking up your own threads waiting on it).

Tuning knobs: error‑rate threshold (e.g. >50%), a minimum request volume before tripping (don't trip on 1 of 1), the open timeout (cool‑down), and how many half‑open trials to admit.

Graceful / Service Degradation

When you can't serve the full experience, serve a reduced one instead of an error. Design the fallbacks up front so degradation is a deliberate mode, not a crash.

Chapter recap