The Pager Didn’t Go Off Until Twitter Did: Automated Incident Detection That Actually Cuts MTTD
If your detection is based on dashboards and vibes, you’re already late. Here’s how to build leading-indicator detection tied to SLOs, triage, and rollout automation so you catch incidents before customers do.
If your alert can’t answer “is the user impacted?” and “what should I do next?”, it’s not incident detection—it’s telemetry cosplay.Back to all posts
Key takeaways
- MTTD drops when you alert on leading indicators (burn rate, saturation, queue depth, tail latency), not vanity metrics (CPU averages, request counts).
- SLO burn-rate alerts are the fastest path to objective detection—and the easiest way to stop paging on nonsense.
- Telemetry isn’t useful until it’s tied to triage: ownership, runbooks, and automated context (deploy SHA, feature flag state, top offenders).
- Progressive delivery (canary + automated analysis) turns detection into prevention by halting/rolling back bad releases automatically.
- Run a lightweight code + telemetry audit before scaling detection; most teams are missing high-cardinality hygiene, semantic conventions, or deploy correlation.
Implementation checklist
- Define 1–3 user-journey SLOs per critical service (availability + latency).
- Implement multi-window, multi-burn-rate alerts for each SLO in `PrometheusRule`.
- Add leading indicators: queue depth, saturation, tail latency, dependency error rate, and retry rate.
- Standardize telemetry with OpenTelemetry semantic conventions and include `service.version` / git SHA on every signal.
- Attach alerts to an owner, a runbook, and a single first action (page, ticket, or auto-rollback).
- Gate rollouts with canary analysis using Prometheus queries; auto-rollback on sustained burn or tail spikes.
- Review alert outcomes weekly: false positives, missed incidents, and time-to-human-diagnosis (not just MTTD).
Questions we hear from teams
- What’s the fastest way to reduce MTTD without a full observability rebuild?
- Start with 1–2 critical user journeys, define SLOs, and implement burn-rate alerting with strict routing (page vs ticket). Then add deploy correlation (`service.version`/git SHA) so alerts immediately point at the likely change.
- Should we alert on CPU, memory, and disk at all?
- Yes, but treat them as supporting signals. Page on user-impacting SLO burn or clear saturation thresholds (e.g., DB connection pool exhaustion). Infrastructure metrics are better used for diagnosis and capacity planning than primary incident detection.
- How do you avoid alert fatigue when adding more detectors?
- Force every alert to map to exactly one of: page, ticket, or automation. Require a runbook URL and an owner. Delete or downgrade any alert that repeatedly pages without user impact.
- We use AI to generate a lot of code—does that change detection strategy?
- The strategy is the same, but the failure modes shift: inconsistent instrumentation, missing timeouts, high-cardinality labels, and “helpful” retry loops are common in AI-generated code. A targeted audit of telemetry quality and resilience patterns pays off fast.
Ready to modernize your codebase?
Let GitPlumbers help you transform AI-generated chaos into clean, scalable applications.