How do we pick SLOs if we don’t have great historical data?

Start with your most critical user journeys (checkout, login, API POST /orders). Choose a conservative 99% target and instrument SLIs you can query today (error rate, latency p95/p99). Iterate monthly. The biggest win is moving to burn-rate style alerts, not picking the perfect number on day one.

Won’t automation cause more incidents if it acts on bad alerts?

Scope automation to high-confidence remediations with clear rollbacks (scale a deployment, pause a rollout, toggle a feature flag). Gate actions behind multi-window conditions and short ‘for’ durations, and always notify the on-call with exactly what changed.

We’re on Datadog/New Relic, not Prometheus—does this still apply?

Yes. The principles are vendor-agnostic: compute burn rates, enrich with ownership, group and inhibit alerts, and wire notifications to triage and rollout APIs. Datadog monitors and event pipelines support the same patterns; so do New Relic workflows and Honeycomb triggers.

How do we handle noisy flapping alerts during incidents?

Use Alertmanager grouping and inhibition to collapse downstream symptoms. Increase group_wait to 30–60s to coalesce bursts. Add rate-of-change or consecutive evaluation windows so a single blip doesn’t page. After the incident, run a noise retro and fix or delete the loudest rules.

Reliability-observability · Oct 6, 2025 · 9 minute read

Stop Paging the Whole Org: Intelligent Alert Routing That Predicts Incidents and Drives Rollbacks

Cut 50–70% of pages by routing on leading indicators, not vanity metrics—and wire those alerts to triage bots and safe rollbacks.

Alex Ramirez

Principal Reliability Engineer, GitPlumbers

20 years in the trenches building and rescuing systems at scale—ex-Netflix SRE, led observability at a unicorn SaaS, and helped teams from seed-stage to Fortune 100 slash MTTR without paging the whole org.

You don’t fix alert fatigue with “Do Not Disturb.” You fix it by making alerts the actuator in your control loop.

Back to all posts

Stop Paging the Whole Org: Intelligent Alert Routing That Predicts Incidents and Drives Rollbacks

Key takeaways

Implementation checklist