What’s a good starting set of leading indicators per service?

p99 latency, error-rate anomaly (relative change), retry rate, internal queue depth/age, dependency 5xx, DB connection pool saturation, and for event-driven systems, consumer lag growth rate. Add runtime-specific ones (GC pause p95/p99 for JVM, FD usage, CPU steal).

How do I prevent alert fatigue when adding more signals?

Group alerts by service and page on budget burn or multi-signal conditions. Use Alertmanager grouping and deduplication, and route WARN to Slack while PAGE only on sustained leading indicators that correlate with user-impact or SLO burn.

We’re on ECS/Lambda, not Kubernetes. Does this still apply?

Yes. Replace Argo Rollouts with CodeDeploy blue/green + CloudWatch Alarms or Flagger on App Mesh. Same Prometheus/OpenTelemetry metrics, same decision trees, different controllers.

Who owns the playbooks—SRE or product teams?

Service-owning teams own their playbooks. SRE provides the framework: templates, tooling, routing, and quality bar. Make playbook PRs part of the service’s definition of done.

How do we test playbooks without breaking prod?

Use shadow traffic and staged canaries. Chaos test in staging with synthetic load. In prod, run controlled fault injections (low blast radius) during low traffic, paired with automatic rollback and SLO burn alerts.

Reliability-observability · Oct 28, 2025 · 9 minute read

The Playbook Problem: Building Incident Response That Scales Across Teams (And Predicts the Blast Before It Happens)

Stop paging on vanity metrics. Start wiring leading indicators to triage and rollout automation so incidents fix themselves—or never happen.

Back to all posts

The Playbook Problem: Building Incident Response That Scales Across Teams (And Predicts the Blast Before It Happens)

Key takeaways

Implementation checklist