What’s the difference between a runbook and a playbook?

Playbooks describe the process (roles, comms, timelines). Runbooks are technical, single-issue, and executable—focused on specific alerts and the exact mitigation/triage steps. You need both, but runbooks shrink MTTR.

How do we keep runbooks from rotting?

Version them with the service, review via PR, and validate them in monthly game days. Any step that breaks in rehearsal gets fixed that day. Tie alerts to specific runbook anchors so edits are obvious when links break.

Which tools do you recommend to start?

Prometheus + Alertmanager for signal; OpenTelemetry for traces; Argo Rollouts or Flagger for progressive delivery; PagerDuty or Opsgenie for paging; Grafana for viz; Sloth to generate SLO/burn rate rules; a simple Slack bot for /triage. Keep it boring and standard.

Do we need chaos tooling to run game days?

Nice to have, not required. Start with `toxiproxy`, `stress-ng`, and traffic generators. Graduate to LitmusChaos or Gremlin when you’re ready for orchestrated scenarios and blast-radius controls.

Reliability-observability · Oct 17, 2025 · 10 minute read

The Runbook-Driven Game Day That Cut MTTR From 72 Minutes to 14

Stop worshiping dashboards. Wire leading indicators into runbooks and progressive delivery so incidents fix themselves—or at least page the right human with the right commands.

Back to all posts

The Runbook-Driven Game Day That Cut MTTR From 72 Minutes to 14

Key takeaways

Implementation checklist