Scenario Playbooks

CHAPTER 4

Scenario Playbooks

The classic on-call interview questions: “latency doubled, what do you do?”, “a provider is failing”, “traffic just went 10×”. The winning answer is never a single trick — it’s a calm, repeatable loop. Mitigate first, root-cause later, and always quantify the blast radius before you touch anything.

Latency spike Provider outage 10× traffic Incident response

Interviewers aren’t testing whether you memorized a fix. They’re watching how you think under pressure: do you jump to a random guess, or do you narrow the problem, protect users, and reason from “what changed?” Every scenario below reuses one framework — learn it once, apply it everywhere.

The universal framework

Mitigate first, root-cause later

When a page fires, your first job is not to understand the incident — it’s to stop the bleeding. Users don’t care why the service is slow; they care that it’s slow right now. So the loop is optimized to restore service quickly, then investigate calmly once the pressure is off.

The one habit that separates seniors from juniors: before doing anything, quantify the blast radius and ask “what changed recently?” — deploys, config, feature flags, a traffic shift, or a dependency. The overwhelming majority of incidents trace back to a recent change.

Detect Assess blast radius Hypothesize Mitigate Verify Root-cause Prevent

Detect. An alert, an SLO burn, or a customer report tells you something is wrong. Acknowledge the page and open the incident channel so responders converge in one place.
Assess blast radius. Quantify it: what percentage of requests are affected, which region/AZ, which endpoint, which customers/tenants? A 2% error rate on one endpoint in one region is a very different incident from a global outage — and it changes how aggressively you mitigate.
Form hypotheses. Start from “what changed in the window?” — recent deploys, config pushes, flag flips, traffic pattern shifts, or a degraded dependency. Rank hypotheses by likelihood, cheapest-to-check first.
Mitigate. Restore service with the least risky lever: roll back the suspect change, shed / rate-limit load to protect the core, or fail over to a healthy region/replica/provider. You do not need root cause to mitigate.
Verify. Watch the SLIs recover. If the mitigation didn’t move the graph, revert it (don’t leave stacked half-fixes) and try the next hypothesis. One change at a time.
Root-cause. Now that users are safe, investigate calmly with full telemetry — logs, traces, metrics — and confirm the actual cause instead of the symptom.
Prevent. Blameless postmortem, concrete action items with owners and dates: better alerting, guardrails, automated rollback, capacity, or a fixed query. Close the loop so this class of incident can’t recur silently.

The incident loop — the same seven steps power every scenario below.

The two questions that solve most incidents

“How big is it?” (blast radius) and “What changed?” (deploys / config / flags / traffic / dependencies). Ask these out loud in the interview — it shows structured thinking before you’ve named a single fix.

Scenario A · Latency doubled

Interviewer

“Your service’s p99 latency just doubled. Nothing is erroring — it’s just slow. Walk me through what you investigate and what you do.”

First reflex: don’t guess. Confirm and quantify, then work the “what changed” list before you go spelunking in the database. Latency is a symptom with a dozen causes — the playbook narrows it fast.

Hypotheses → signal → likely fix

Hypothesis	Signal to check	Likely fix
Recent deploy / release	Deploy timeline vs. the latency inflection; canary/baseline metrics diverging	Roll back the release or the canary
Downstream dependency slow	Per-dependency p99 & error rate; a fan-out call whose tail exploded	Circuit-break / fail over / add timeout + retry budget
Database slow	Slow-query log, lock/row contention, connection-pool saturation, replica lag	Fix/kill the query, add an index, raise pool, add capacity
Cache degraded	Hit-ratio drop, evictions spiking, a cold/flushed cache, hot-key stampede	Warm cache, add request coalescing, size up
Resource saturation	Host CPU, memory pressure, GC pause time, thread-pool/queue depth	Scale out / restart leaking hosts / tune GC
Traffic / payload change	QPS up, payload sizes up, a new expensive query pattern or hot tenant	Rate-limit / shed the abusive pattern
Noisy neighbor / network	Packet loss, retransmits, one bad AZ/host, cloud provider event	Drain the host/AZ; fail over to healthy infra

Step-by-step playbook

Confirm & quantify. Which endpoints? Is it p50 vs p95 vs p99 — did the whole curve shift up (systemic: saturation, dependency) or only the tail (queuing, GC pauses, a slow shard, lock contention)? Tail-only vs. whole-curve points you at very different causes.
Check blast radius. One region/AZ, one customer, one code path — or everything? Scope tells you whether to fail over a region or roll back a global change.
Check recent changes FIRST. Deploys, config pushes, feature flags in the time window. If the latency step lines up with a release, roll it back before doing anything cleverer — it’s the highest-probability cause and the cheapest mitigation.
Inspect dependency latency. Walk the fan-out: a single slow downstream call inflates your p99. Look at each dependency’s p99 and error rate; a circuit breaker + timeout keeps one slow dependency from dragging the whole request down.
Inspect DB & cache. Slow queries, replica lag, lock contention, connection-pool saturation, and cache hit-ratio drops are the usual heavy hitters. A dropped hit ratio quietly pushes 10× load onto the database.
Check saturation. CPU, memory, GC pause time, thread pools, connection counts, queue depth. Latency climbing with utilization near 100% means you’re out of headroom — scale out.
Decide & mitigate. Pick the lever: roll back (a change did it), scale out (saturation), or fail over (bad region/dependency). Restore first, verify the graph recovers, then do the RCA.

Latency triage — cheapest, highest-probability checks first: change → dependency → saturation → data layer.

Say this: “I confirm which endpoints and whether it’s the tail or the whole curve, then check what deployed in that window — a rollback is usually the fastest fix. If nothing shipped, I walk the fan-out for a slow dependency, check DB/cache for slow queries and hit-ratio drops, and look at saturation. I mitigate with a rollback, failover, or scale-out, verify the graph recovers, and only then do the full RCA.”

Scenario B · A provider is failing

Interviewer

“You depend on an external model/provider (say, an LLM API or a payment gateway). One of them starts timing out and throwing 5xx. How do you keep your service healthy?”

This is the isolation-and-reroute story. A single sick dependency must not take you down with it. The tools are a circuit breaker, fallback routing, and graceful degradation — all driven by per-provider telemetry.

🔌

Circuit breaker

Trip when a provider’s error rate or timeout rate crosses a threshold. Stop hammering the sick dependency — fail fast instead of piling up slow, doomed calls.

🔀

Fallback routing

Route to a healthy provider, region, or a cheaper/smaller model. Weighted routing lets you shift traffic gradually rather than slamming the backup.

📉

Graceful degradation

If no healthy provider exists, degrade: queue and retry later, serve a smaller model, or a cached/canned response. Partial service beats a hard failure.

🛡️

Protect the error budget

Don’t let a third party burn your SLO. Isolate its failures so a provider outage costs you a degraded feature, not a violated availability target.

📊

Per-provider telemetry

Track success rate, latency, and cost per provider. You can’t route around a failure you can’t see — these SLIs drive the breaker and the routing weights.

🔁

Retry with a budget

Retries with backoff + jitter help transient blips, but cap them with a retry budget so you don’t amplify an overloaded provider into a retry storm.

Breaker trips on A, weighted routing shifts to healthy B, C absorbs spillover — the outage becomes a degraded feature, not a page-one failure.

Step-by-step playbook

Detect via per-provider SLIs. Alert on success rate / latency for each provider independently. A blended metric hides which dependency is sick.
Open the breaker on the bad provider. Trip on error% or timeout rate so you stop sending doomed traffic and fail fast — this alone protects your latency.
Shift traffic to healthy providers. Use weighted/fallback routing to move load onto B (and C as spillover). Shift gradually so you don’t overload the backup.
Degrade gracefully if all are unhealthy. Queue requests for async processing, drop to a smaller/cheaper model, or serve cached/canned responses. Keep the core experience alive at reduced quality.
Communicate. Update the status page and stakeholders — “feature X is degraded, we’ve rerouted.” Set expectations before customers file tickets.
Recover with half-open probes. Periodically let a trickle of traffic test the recovered provider (half-open). If it succeeds, ramp weights back up; if not, re-open the breaker.
Postmortem. Add the missing timeout, tune breaker thresholds, verify the fallback is actually tested, and consider a second provider if you had none.

Say this: “I isolate the failure so it can’t take me down. Per-provider telemetry detects it, a circuit breaker trips on the bad provider, and weighted fallback routing shifts traffic to a healthy one. If everything’s degraded I fail soft — queue and retry, a smaller model, or a cached response — protect the error budget, update the status page, and recover with half-open probes.”

Scenario C · Traffic jumped 10×

Interviewer

“A launch (or a viral moment) just pushed your traffic to 10× normal. The service is buckling. What do you do to keep it up — right now, and then durably?”

Two clocks are running: survive the spike now (protect the core so it doesn’t topple), then scale durably. The order matters — you shed and cache before you can add capacity, because scaling isn’t instant and an overloaded service can’t even serve health checks.

10×

incoming traffic

Shed

protect the core first

↔️

stateless = scalable

usual bottleneck

The levers

↔️

Horizontal auto-scaling

Add stateless instances behind the load balancer. This only works if the tier is stateless — no sticky local state, sessions in a shared store. Stateless is the whole game for scaling.

📥

Queue to absorb bursts

Put a queue in front of expensive/write work. It smooths spikes: producers enqueue fast, workers drain at a sustainable rate instead of the DB being hit at peak QPS.

⚡

Caching & CDN

Raise the hit ratio, push static/read content to a CDN, and coalesce duplicate requests. A good cache turns 10× origin load into 10× edge load and ~1× origin.

🚦

Rate limit / shed load

Admission control + load shedding drop or throttle excess so the core stays alive. Serving 80% of users well beats serving 100% a crash.

📚

Read replicas

Fan read-heavy load out to replicas and keep the primary for writes. Most workloads are read-dominated, so this buys a lot of headroom quickly.

🗂️

Async & precompute

Move non-critical work off the request path (emails, analytics, thumbnails) and precompute hot results. Do less work synchronously per request.

The database is usually the bottleneck

The stateless tier scales out trivially; the database — especially writes — does not. You can add read replicas and cache reads, but write throughput is hard to scale. That’s why you put a queue in front of writes and reach for sharding/partitioning when a single primary runs out of room. Assume the DB is where 10× hurts most.

Absorb reads at the edge, buffer writes through a queue, scale the stateless tier — and shield the primary DB, the part that scales worst.

Step-by-step playbook

Protect first. Turn on / tighten rate limiting and load shedding so the core survives while you react. An overloaded service that stays up beats one that falls over trying to serve everyone.
Scale the stateless tier. Add instances behind the LB (auto-scaling if configured). This is your fastest real capacity gain — as long as the tier holds no local state.
Offload reads. Push reads to cache/CDN and read replicas, raise hit ratios, and coalesce duplicate requests. This removes the most load for the least effort.
Buffer writes. Put a queue in front of write-heavy work and let async workers drain at a sustainable rate, decoupling the spike from the database.
Watch the DB. It’s the usual bottleneck — monitor connections, replication lag, write latency, and locks. If the primary is the wall, that’s your durable-capacity problem to solve next.
Plan durable capacity. Once stable, right-size auto-scaling limits, add replicas/sharding, tune cache TTLs, and load-test to the new peak so the next spike is a non-event.

✅ Do

Shed/rate-limit before you fall over — keep the core alive.
Scale the stateless tier; keep sessions in a shared store.
Cache aggressively and coalesce duplicate requests.
Put a queue in front of writes; drain async.
Assume the DB is the bottleneck and protect it.

🚫 Don't

Try to serve 100% and crash serving 0%.
Add instances to a tier that holds local state — it won’t scale.
Point 10× reads straight at the primary database.
Retry aggressively into an overloaded backend (retry storm).
Assume auto-scaling is instant — warm-up and DB limits still bite.

Say this: “First I protect the core — rate-limit and shed so it doesn’t topple. Then I scale the stateless tier horizontally, offload reads to cache/CDN and replicas, and put a queue in front of writes so async workers drain at a safe rate. The database is the usual bottleneck, so I watch it closely and plan sharding/replicas for durable capacity once we’re stable.”

Handling any incident well

The mechanics of a fix matter less than how you run the incident. Interviewers love candidates who talk about communication, discipline, and learning — not just clever mitigations.

Practice	Why it matters
Declare an incident & open a channel	One place for responders to converge — no scattered DMs. Reduces duplicated effort and lost context.
Name an Incident Commander	The IC coordinates, delegates, and decides — they don’t debug. Clear ownership prevents “everyone / no one” chaos.
Update the status page	Tell customers early. Proactive communication cuts ticket volume and preserves trust more than a fast-but-silent fix.
Don’t fix blind	Form a hypothesis, check a signal, then act. Random changes under pressure make things worse and muddy the timeline.
One change at a time	Change → observe → decide. If you stack changes you can’t tell what helped, and you can’t cleanly revert.
Keep a timeline	Timestamped log of what you saw and did. It powers the postmortem and prevents “what did we already try?”.
Blameless postmortem + action items	Fix the system, not the person. Concrete owned action items with dates are how the same incident stops recurring.

Mitigation over diagnosis, every time

If a rollback or failover restores service, do it now — even if you’re not 100% sure it’s the cause. You can always investigate a healthy system. You can’t investigate a happy customer who already left.

On-call cheat sheet

Keep these one-liners in your head when the pager fires

Mitigate first, root-cause later — restore service, then investigate.
Ask the two questions: “How big?” (blast radius) and “What changed?” (deploy / config / flag / traffic / dependency).
Latency doubled? Check the deploy window first, then dependency p99, then DB/cache, then saturation.
Whole curve up = systemic (saturation/dependency). Tail only = queuing, GC, locks, a slow shard.
Provider failing? Circuit-break it, fail over to healthy, degrade gracefully if all sick.
10× traffic? Shed to protect the core, scale the stateless tier, cache reads, queue the writes.
The database is the usual bottleneck — replicas for reads, queue + sharding for writes.
One change at a time. If it didn’t help, revert it before trying the next thing.
Open an incident channel, name an IC, update the status page, keep a timeline.
Close with a blameless postmortem and owned, dated action items.