What’s the minimum set of leading indicators per service?

For most HTTP services: (1) SLO burn rate (fast+slow window), (2) p95 latency, (3) saturation signal (thread/conn pool, GC, pod restarts), (4) dependency error/latency, and (5) queue depth/lag if asynchronous work exists. Start with 3–5 page-worthy alerts max.

How do we keep playbooks from drifting across teams?

Version them with the service in Git, require `runbook_url` on all paging alerts, and review playbook correctness as part of the post-incident process. If the alert fired and the runbook wasn’t useful, treat that as a bug.

We have Prometheus, but not SLO burn rate recording rules. Is that required?

You can start with error ratio + latency thresholding, but burn rate is the scalable model because it normalizes by traffic and ties directly to error budgets. Add recording rules early; it pays back quickly in reduced noise and faster decisions.

How do we tie deploys to incidents without buying a fancy platform?

Annotate Grafana dashboards with deploy events, include `deploy_url` in alerts, and ensure traces/logs include `deploy.version` (git SHA). Even a simple link to ArgoCD/CI runs cuts triage time dramatically.

Can we automate rollback safely?

Yes—when rollback is part of the deployment system (e.g., Argo Rollouts) and is gated by the same telemetry you trust for paging (burn rate, latency). Don’t auto-rollback on CPU or generic health checks; that’s how you get flapping deployments.

Reliability-observability · Jan 7, 2026 · 8 minute read

Your Incident Playbooks Don’t Scale—Until You Treat Alerts Like APIs

Multi-team incident response breaks when every squad invents its own rituals. The fix isn’t “more dashboards.” It’s playbooks that bind leading indicators to triage and rollout automation.

Back to all posts

Your Incident Playbooks Don’t Scale—Until You Treat Alerts Like APIs

Key takeaways

Implementation checklist