What’s the minimal viable setup to cut MTTD this quarter?

Pick one high-traffic service. Define a 99.9% availability SLO for a single golden path. Ship a fast/slow burn-rate alert, a DB pool saturation alert, and a consumer lag-growth alert. Enrich pages with service + SHA + runbook. Add an Argo Rollouts AnalysisTemplate to gate canary at 10% and 25%. That alone will shave minutes off detection.

How do I keep alerts from waking up the world?

Multi-window burn-rate alerts, ownership labels, and hard routing rules. Page the team that owns the SLO; ticket everyone else. De-duplicate in Alertmanager. Measure false-positive rate and delete or tune noisy rules weekly.

We use Datadog/New Relic, not Prometheus—does this still apply?

Yes. The patterns (SLO, burn-rate, saturation, queue growth, gating) work anywhere. Datadog monitors can implement burn rate. Flagger supports multiple providers. The key is the signals, not the tool logo.

Our codebase is partially AI-generated and lacks telemetry. Where do we start?

Do a vibe code cleanup: add OTel middleware for traces and RED metrics, instrument resource pools, and emit structured error reasons. Bake metric names and labels into a shared library so teams can’t drift. You can bolt this on in days, not months.

How do we prove this works to the business?

Track MTTD, alert-to-mitigation time, and error-budget spend. After two sprints, you should see MTTD fall by 50–80% on the target service and fewer user-visible incidents per deploy. Tie that to revenue protection (checkout success rate, ride match rate, etc.).

Reliability-observability · Nov 23, 2025 · 9 minute read

Killing MTTD: Leading-Indicator Alerts That Roll Back Before Users Notice

Stop paging on vanity dashboards. Build predictive signals tied to SLOs, auto-triage, and gated rollouts that catch incidents early and fix them faster than the tweetstorm starts.

Back to all posts

Killing MTTD: Leading-Indicator Alerts That Roll Back Before Users Notice

Key takeaways

Implementation checklist