On call for a 3 person team: design alerts that page less, write runbooks, automate remediation, and degrade gracefully so 3am pages become 9am tickets.
At 3:14 a.m. my phone screamed itself off the nightstand, and by the time my laptop booted the dashboard was already green. The "incident" was a health check that had flapped because a third-party API took 1.2 seconds instead of its usual 400ms — then recovered on its own, the way it always does. I sat there in the dark, heart still going, staring at a graph that had fixed itself. The system had woken a human to watch it get better.
That night I stopped treating on-call as a scheduling problem and started treating it as a design problem. When you're three engineers and one of them is asleep — and the other is on a plane, and you're the third — you cannot out-staff your alerts. You have to build a system that pages less, degrades gracefully, and tells the person who does wake up exactly what to do. This is the small-team on-call and incident response strategy that actually works when the rotation is a fiction: how to cut alert fatigue, write runbooks a groggy human can follow, automate the first remediation step, and protect the people holding it all together.
Every on-call guide you'll read was written by someone with a platform team. The math they assume is simple: enough humans that any one person is on the hook maybe one week in six, with a secondary to escalate to, and a follow-the-sun rotation so Sydney covers the small hours for London.
We have none of that. Do the arithmetic on a three-person on-call rotation:
So the enterprise moves — bigger rota, tighter SLAs, a dedicated incident commander — don't scale down. They assume the one resource we're shortest on: people. The only lever a small team can actually pull is the rate of pages. Everything in this post is downstream of one rule: the cheapest incident is the one that never pages a human.
I learned this the expensive way. Early on at Shpper I wired up alerts the way I'd seen at bigger shops — CPU, memory, latency, error rate, queue depth, all with thresholds copied off a blog post. Within two weeks we were getting six to ten pages a night, almost all self-resolving. Then one Thursday a real one landed — the payments webhook was silently dropping events — and I slept through it, because I'd started swiping the pager away half-asleep like an alarm clock. We found out from a customer email the next morning.
That's the trap, and it has a name: alert fatigue. People mute notifications, and now the alerts are still noisy and nobody's watching. An alert nobody trusts is worse than no alert, because it costs you the one that mattered. On a small team you feel this faster than a big one does, because there's no fresh pair of eyes rotating in every week to reset the tolerance. The same three people absorb every false page until they stop reacting.
The single highest-leverage change we made was to split "something is off" from "wake a human up." Most monitoring conflates them. They are completely different questions.
My test for a paging alert is one sentence: would I be angry if this woke me and I couldn't do anything about it? If yes, it's not a page. It's a ticket, a dashboard, or a Slack message that waits until morning.
Concretely, we sort every signal into three severity tiers:
The other rule is the one most teams get backwards: page on symptoms, not causes. CPU at 90% is a cause and it might be totally fine — maybe you're just busy, maybe the garbage collector is doing its job. What you actually care about is whether users can do the thing. A cause-based alert fires on a possible problem; a symptom-based alert fires on a real one. So we alert on the outcome — the error rate on a user-facing route, the success rate of a critical transaction, the freshness of a queue users depend on:
# Alertmanager: page on user-visible symptoms, not machine internals.groups: - name: user-facing rules: - alert: CheckoutErrorRateHigh # >5% of checkout requests failing over 5 min — a real user impact. expr: | sum(rate(http_requests_total{route="/checkout", code=~"5.."}[5m])) / sum(rate(http_requests_total{route="/checkout"}[5m])) > 0.05 for: 5m labels: severity: page annotations: summary: "Checkout failing for >5% of users" runbook: "https://wiki.internal/runbooks/checkout-5xx"Two details there matter more than the query. The for: 5m clause means the condition has to hold for five minutes — this alone killed most of our flapping, because transient blips recover before the timer fires. It's the single cheapest fix for a noisy pager: most self-resolving incidents resolve inside a couple of minutes, so a short hold window filters them out without you writing a line of remediation code.
And every paging alert carries a runbook link. If an alert can't point to a runbook, it isn't ready to page anyone. That's a hard gate for us — a page without a runbook is just a notification that you failed to prepare.
We also set a deliberate ceiling: no more than roughly two pages per night on a healthy system, ever. If we breach that, the bug isn't in production — it's in our alert config, and fixing the alert becomes the next day's top priority. Treat page volume as a first-class service-level objective for your own sanity. An SLO isn't only about uptime; the humans running the system have a reliability budget too, and burning it is as real an outage as any 500.
One subtlety worth calling out: when you first tighten alerts you will feel exposed. The instinct is that fewer alerts means you'll miss something. In practice the opposite happens — a pager that fires twice a night gets read every time, and a pager that fires ten times a night gets read zero times. Fewer, better alerts is higher coverage, not lower.
Here's the reframe that changed how our team operates: the deliverable of an incident is not the fix. It's the runbook. The fix stops the bleeding tonight; the runbook is what lets a groggy, half-asleep engineer — possibly the one who didn't build that service — resolve the same class of problem in four minutes next time without waking anyone else.
A good small-team runbook is not a wiki essay. Nobody reads three pages of prose at 3 a.m. It's a checklist a tired person can follow with their eyes half-open. Ours have exactly four parts:
## Runbook: Checkout 5xx spike### Confirm- Dashboard: https://grafana.internal/d/checkout- Is the payment provider up? -> https://status.stripe.com### Diagnose (run this first)
kubectl logs -l app=checkout --since=10m | grep -c "gateway_timeout"
### Fix- If gateway_timeout > 0 and provider is degraded: flip the feature flag to queue orders instead of failing. `./scripts/toggle.sh checkout_queue_mode on`- This returns "Order received, confirming shortly" to users instead of an error. Orders drain automatically when the provider recovers.### If that didn't work- Grab the last 30 min of checkout logs and the flag state.- Wake the on-call. This is now a real escalation.
Notice the fix isn't "debug the payment integration at 3 a.m." It's "flip a flag that makes the problem wait until morning." Hold onto that instinct — it's the whole game for a team this size, and the next two sections are just it applied harder.
One more thing we're strict about: runbooks decay. A command that worked last quarter breaks when the service moves clusters or the flag gets renamed. So the postmortem step for every paged incident includes "did the runbook actually work?" If a step was wrong, fixing it is part of closing the incident, not a someday-maybe. A runbook you can't trust is as dangerous as an alert you can't trust — it sends a tired person down a dead end while the clock runs.
Most teams automate detection — monitoring, alerting, dashboards — and then hand a human every response. For a big team that's fine; there's always a human. For us, the response is the expensive part, because the human is asleep.
So we push automation one step further, into the first remediation. This is the small-team version of self-healing infrastructure: for any incident where the safe response is deterministic, the machine should try it before it pages.
The auto-rollback is the one I'd fight hardest to keep. Most of our would-be-3 a.m. incidents were bad deploys, and a deploy is the one failure with a known-good previous state sitting right there. Rolling back is nearly always safe because you already ran that version in production five minutes ago.
# CI: gate the deploy on real health, roll back automatically if it fails.- name: Post-deploy health gate run: | ./scripts/wait-healthy.sh --url "$PROD_URL/health" --timeout 120 || { echo "Health check failed — rolling back." ./scripts/rollback.sh --to "$PREVIOUS_RELEASE" ./scripts/notify.sh "#alerts" "Auto rolled back $GIT_SHA. Prod is on $PREVIOUS_RELEASE." exit 1 }The health gate has to check something user-visible, not just that the process booted — a container can start cleanly and still serve 500s because a config value is missing. Point wait-healthy.sh at an endpoint that exercises a real dependency (hits the database, checks the payment client can authenticate) so a green check actually means the app works.
The rule of thumb: if the response to an alert is always the same three commands, those commands belong in a script, not in a human's muscle memory at 3 a.m. Every runbook step that never varies is a candidate for automation. The runbook is where automation goes to be born — you write the manual version first, run it a few times to prove it's correct, then promote the deterministic parts to code. That ordering matters: automating a remediation you've never run by hand is how you build a robot that confidently makes the outage worse.
This is the mindset shift that made on-call survivable for us: your job at 3 a.m. is not to fix the system. It's to make the failure boring enough that fixing it can wait until 9.
A hard failure — checkout throws a 500 — forces an immediate response. A soft failure — checkout says "we've received your order, confirmation shortly" and quietly queues it — buys you eight hours of sleep. Same underlying outage; wildly different on-call cost. Graceful degradation is how you convert a page into a ticket, and it's the highest-return reliability work a small team can do because it directly buys back sleep.
We design the important paths so each dependency has a defined "the world is on fire" mode:
The pattern in code is always the same three moves: a tight timeout, a catch that logs instead of throwing, and a sane fallback value.
// Fail soft: a degraded feature should never take down the whole screen.Future<List<Product>> loadRecommendations(String userId) async { try { return await recommendationService .fetch(userId) .timeout(const Duration(milliseconds: 800)); } on TimeoutException { // The recommender being slow is not an emergency. // Show something reasonable and let someone look tomorrow. _log.warn('recs timed out, serving fallback'); return _popularItemsCache.current(); } on RecommendationException catch (e, s) { _log.error('recs failed, serving fallback', e, s); return _popularItemsCache.current(); }}The timeout is doing the heavy lifting. Without it, a slow dependency doesn't fail — it hangs, holding connections until the whole screen locks up and you get the worst kind of page: everything's technically "up" but nothing works. This is the cascading-failure pattern, and it's brutal precisely because no single component looks broken. Aggressive timeouts plus a sane fallback turn a cascading failure into a logged warning. I'd rather serve slightly stale recommendations for six hours than get woken to restart a service.
A word on picking the timeout: it should be tied to what the user will tolerate, not to how slow the dependency can theoretically be. If a screen has to render in a second, a recommendation call that can't finish in 800ms is already useless to you — so cut it off there and move on. Timeouts sized to the dependency's worst case are how you end up hanging for thirty seconds waiting for something you were never going to be able to show.
The full ceremonial postmortem — the five-whys workshop, the multi-page document, the cross-team review — is a luxury good. On a three-person team, if the process costs more than the incident, nobody does it, and you learn nothing. A postmortem that never gets written teaches you exactly as much as no postmortem at all.
So we run a stripped-down version. Any incident that paged a human gets fifteen minutes, async, in a shared doc, answering four questions:
"Blameless-ish" is deliberate. On a team this small, "who caused it" is almost always known and almost always one of three people — pretending otherwise is silly. What we're strict about is that the fix targets the system, never the person. "Shakib pushed a bad migration" is a fact. The action item is never "Shakib should be more careful." It's "migrations run behind a health-gated deploy that auto-rolls-back" — which is exactly the automation two sections up. Most of our best reliability work started as a postmortem action item.
The honest failure mode here is skipping question 4. It's tempting to close the doc once prod is green. But the whole point is that each incident should make the next one less likely to page. If your postmortems don't shrink your future page volume, they're just paperwork. The test of a good postmortem isn't the quality of the writeup — it's whether the same alert fires again next month.
The part nobody writes about: a bad on-call setup doesn't just cost sleep, it costs people. On a three-person team, losing one engineer to burnout isn't a staffing hit, it's a third of your capacity and most of your morale. Protecting the humans is not soft stuff — it's the same reliability problem pointed at a different system. Your teammates are a dependency too, and they degrade under sustained load exactly like a service does.
What we actually do:
notify-tier alert at night, "seen, it can wait till morning" is a complete and correct response. We had to say this out loud, because the instinct is to fix everything the moment you see it.The uncomfortable truth: if your on-call only works because one person is quietly absorbing all the pain, it doesn't work. It's just failing slowly, in a way that won't show up on a dashboard until that person quits. And when they do, you don't lose a third of your on-call coverage — you lose a third of the institutional knowledge about why every alert exists, which is far more expensive to rebuild.
You can't feel your way to a healthy on-call — "it seems quieter lately" is not data. But you also don't need an SRE metrics stack. Three numbers, reviewed monthly, tell the whole story for a small team:
| Metric | What it really tells you | Healthy trend |
|---|---|---|
| Pages per week (esp. overnight) | Whether your alerts respect human sleep | Flat-low or falling |
| % of pages that were actionable | Whether you're crying wolf | Rising toward ~100% |
| % of pages with a working runbook | Whether knowledge is captured or trapped in one head | Rising toward ~100% |
I deliberately don't obsess over MTTR (mean time to resolve) on a team this small — the sample size is too tiny for the average to mean anything, and one weird incident skews a whole quarter. The three above are leading indicators you can actually move, whereas MTTR is a lagging one you mostly just watch.
The one I watch hardest is actionable page percentage. When it drops, it means we've started paging on noise again, and noise is how you train a team to ignore alerts. If nine of every ten pages needed a real human action, the team trusts the pager. If it's one in ten, they've already muted it and you just haven't noticed yet. This single number is the earliest warning that alert fatigue is creeping back in — it moves before the "pages per week" number does, because people stop acting on alerts before they stop receiving them.
On-call for a tiny team isn't a rota problem, it's an engineering problem. You survive by making the system page less, not by being more heroic.
for: hold window to kill flapping. Everything else is a ticket or a log line.