What’s a good starting set of leading indicators?

For web APIs: p95 latency on the hot path, error rate, worker/thread pool queue depth, DB connection pool utilization, Kafka/SQS backlog, CPU throttling ratio, and OOM/restart spikes. Tie them to SLOs instead of static thresholds.

How do we keep playbooks from going stale?

Treat them like code: store in Git, lint in CI (validate schema and external links), and run monthly game days. Expire or fix any playbook step that fails during drills.

Can we do this with Datadog/New Relic instead of Prometheus?

Yes. Use Datadog monitors with multi-window burn-rate formulas and tags for team/service. The same concepts apply—just wire the dashboards, runbook URLs, and rollout metadata into annotations and PagerDuty/Slack.

We tried AI to draft runbooks. Worth it?

Sure—as a first draft. But review for executability and accuracy. We’ve done a lot of vibe code cleanup where AI hallucinated commands or paths. Add validation, dry-run support, and drill before trusting it on-call.

How does this play with feature flags?

Great. Put kill switches and traffic shapers in the mitigate section and expose them as commands. Tie flags to telemetry so you can degrade gracefully during incidents while the canary auto-aborts if metrics fail.

Reliability-observability · Nov 27, 2025 · 10 minute read

Your Incidents Are Predictable: Build Playbooks That Route, Triage, and Roll Back Themselves

Stop paging on vanity graphs. Use leading indicators, wire telemetry to triage, and let rollouts auto-protect you. Here’s how to make incident playbooks that actually scale across teams.

Back to all posts

Your Incidents Are Predictable: Build Playbooks That Route, Triage, and Roll Back Themselves

Key takeaways

Implementation checklist