What’s the difference between a runbook and a playbook?

Playbooks describe the process (roles, comms, timelines). Runbooks are technical, single-issue, and executable—focused on specific alerts and the exact mitigation/triage steps. You need both, but runbooks shrink MTTR.

How do we keep runbooks from rotting?

Version them with the service, review via PR, and validate them in monthly game days. Any step that breaks in rehearsal gets fixed that day. Tie alerts to specific runbook anchors so edits are obvious when links break.

Which tools do you recommend to start?

Prometheus + Alertmanager for signal; OpenTelemetry for traces; Argo Rollouts or Flagger for progressive delivery; PagerDuty or Opsgenie for paging; Grafana for viz; Sloth to generate SLO/burn rate rules; a simple Slack bot for /triage. Keep it boring and standard.

Do we need chaos tooling to run game days?

Nice to have, not required. Start with `toxiproxy`, `stress-ng`, and traffic generators. Graduate to LitmusChaos or Gremlin when you’re ready for orchestrated scenarios and blast-radius controls.

Reliability-observability · Oct 17, 2025 · 10 minute read

The Runbook-Driven Game Day That Cut MTTR From 72 Minutes to 14

Stop worshiping dashboards. Wire leading indicators into runbooks and progressive delivery so incidents fix themselves—or at least page the right human with the right commands.

Avery Quinn

Partner, Reliability & Delivery, GitPlumbers

20 years building and fixing distributed systems at scale. Led SRE and platform teams through two cloud migrations, a microservices detour, and the last three AI hype cycles. Calm during incidents; allergic to vanity dashboards.

If your runbook starts with “Log into the console,” you’ve already lost. Make the computer do the boring parts.

Back to all posts

The Runbook-Driven Game Day That Cut MTTR From 72 Minutes to 14

Key takeaways

Implementation checklist