Your Dashboards Aren’t Detecting Incidents — Your Rollouts Are

The fastest outages are the ones your users detect first

I’ve sat in too many incident reviews where the timeline starts with: “Customer Success pinged us” or “someone saw a spike in 500s in Kibana.” That’s not “observability”—that’s crowdsourced monitoring.

Teams usually respond by adding more dashboards and more alerts. The result is predictable:

• Pager fatigue (alert storms during deploys)
• Vanity metrics (CPU%, request count) that move all the time but don’t predict failure
• MTTD that doesn’t budge because the signal isn’t tied to user impact

Here’s what actually works: build automated detection around leading indicators that predict customer harm, then wire those signals into triage and rollout automation. If your deploy pipeline can ship code, it can also stop shipping bad code.

Leading indicators: what pages you before the outage headline

A lot of orgs still page on “CPU > 80%” because it’s easy. I’ve seen CPU sit at 20% while the system is on fire (deadlocks, downstream timeouts, DB connection pool exhaustion). I’ve also seen CPU at 95% for months because someone oversized the node and forgot to tune requests/limits.

The leading indicators that consistently correlate with incidents:

• SLO burn-rate (error budget consumption velocity)
• Tail latency drift (p95/p99) even when averages look fine
• Saturation (thread pools, connection pools, GC pressure, CPU throttling, memory pressure)
• Queue growth / lag (Kafka consumer lag, SQS age, background job depth)
• Dependency health (timeouts, reset rates, downstream 5xx, DNS failures)

Two practical heuristics that keep you honest:

• If it doesn’t map to user experience or business outcome, it’s probably not page-worthy.
• If it can’t answer “what changed?”, it won’t reduce MTTR even if it reduces MTTD.

Burn-rate alerting: the only “latency spike” alert I still trust

If you want fewer pages and earlier detection, burn-rate alerting is the grown-up move. It’s how mature SRE teams (Google-style) avoid paging on random noise.

Define an SLO (example: 99.9% of requests are successful over 30 days). Then alert when you’re burning error budget too fast.

Prometheus rules example (multi-window, fast + slow burn):

# prometheus-rules.yaml
groups:
- name: checkout-slo
  rules:
  - record: slo:checkout:errors:rate5m
    expr: sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="checkout"}[5m]))

  - alert: CheckoutSLOFastBurn
    expr: slo:checkout:errors:rate5m > (14.4 * (1 - 0.999))
    for: 5m
    labels:
      severity: page
      service: checkout
    annotations:
      summary: "Fast burn: checkout error budget melting"
      description: "5m error rate is too high for 99.9% SLO. Likely customer impact."
      runbook_url: "https://internal/wiki/runbooks/checkout"

  - alert: CheckoutSLOSlowBurn
    expr: slo:checkout:errors:rate5m > (6 * (1 - 0.999))
    for: 30m
    labels:
      severity: ticket
      service: checkout
    annotations:
      summary: "Slow burn: checkout trending towards SLO breach"
      description: "Not paging yet, but this predicts an incident if it continues."
      runbook_url: "https://internal/wiki/runbooks/checkout"

A few notes from scars-earned experience:

• Fast burn pages. Slow burn creates a ticket/Slack thread and gets handled in daylight.
• Use multi-window logic (short + long) if you can. It kills flappy alerts.
• Burn-rate alerts are a leading indicator because they fire when the slope is bad, not when the system is already dead.

Tie telemetry to triage: alerts should answer “what broke, where, what changed?”

The difference between “we detected it” and “we fixed it” is whether the on-call can identify the blast radius in the first 2 minutes.

Three concrete techniques that reduce time-to-triage:

• Stamp deployment metadata into metrics/logs/traces (git_sha, version, environment, rollout_id)
• Include those labels in alerts so you can correlate to deploys
• Link every page to a runbook with the first 5 minutes, not a novel

If you’re using OpenTelemetry, push resource attributes that end up as labels:

# otel-collector.yaml
processors:
  resource:
    attributes:
      - key: service.version
        from_attribute: k8s.pod.labels.app_kubernetes_io/version
        action: upsert
      - key: git.sha
        from_attribute: k8s.pod.labels.git_sha
        action: upsert

exporters:
  prometheusremotewrite:
    endpoint: https://prom.example.com/api/v1/write

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource]
      exporters: [prometheusremotewrite]

Now your burn-rate alert can carry service.version and git.sha. That’s the “what changed?” breadcrumb you want when the page hits.

Also: stop paging without context. If the alert can’t tell you region, dependency, or rollout, it’s noise.

Rollout-aware detection: gate canaries on SLO signals (and auto-abort)

Most real incidents I’ve seen in the last decade weren’t cosmic rays. They were:

• a bad deploy
• a config change
• a dependency behavior change

So make your incident detection rollout-aware. If a canary goes sideways, you shouldn’t be waiting for a human to notice Grafana.

Here’s an Argo Rollouts example that gates a canary on p99 latency and 5xx rate from Prometheus. When it regresses, Argo can automatically abort.

# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 2m}
      - analysis:
          templates:
          - templateName: checkout-slo-guard
      - setWeight: 50
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: checkout-slo-guard
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-slo-guard
spec:
  metrics:
  - name: http-5xx-rate
    interval: 30s
    successCondition: result[0] < 0.002
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          sum(rate(http_requests_total{service="checkout",status=~"5.."}[2m]))
          /
          sum(rate(http_requests_total{service="checkout"}[2m]))

  - name: p99-latency
    interval: 30s
    successCondition: result[0] < 0.8
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[2m]))
          )

This is where MTTD collapses from “minutes” to “seconds” because detection happens inside the rollout, not after users complain.

If you’re doing GitOps with ArgoCD, treat this as first-class config. PR review is where you decide what “safe” means.

Automate the first 60 seconds: pause, rollback, shed load, flip flags

I’ve watched teams build beautiful alerting… and then still spend 20 minutes arguing in Slack about what to do. The trick is to automate the first move.

Pick one or two safe actions per service:

• Pause/abort rollout (best ROI for deploy-caused incidents)
• Rollback to last known good (if your rollback is actually safe—test this)
• Flip a feature flag off (LaunchDarkly, Unleash, or even a homegrown flags service)
• Shed load (rate limiting, circuit breaker, fail open/closed decisions)

A pragmatic pattern is: alert fires → incident created → webhook triggers a “stop the bleeding” action.

Example: trigger an Argo Rollouts abort via CLI from an automation job:

# abort-rollout.sh
set -euo pipefail
ns=prod
name=checkout
kubectl -n "$ns" argo rollouts abort "$name"
kubectl -n "$ns" argo rollouts get rollout "$name" --watch=false

Wire that behind a guardrail (only severity=page, only during an active rollout, require 2 consecutive failures). The goal isn’t to eliminate humans; it’s to stop the blast radius while the human shows up.

At GitPlumbers we often implement this as a small “incident bot” that:

• posts the alert + rollout diff into a Slack channel
• links the runbook
• pauses the rollout automatically if it’s the suspected trigger

It’s boring, repeatable engineering—and it works.

Measure detection quality (and kill the vanity metrics)

If you want MTTD to actually improve, measure the pipeline like a product.

Metrics that matter:

• MTTD: time from first user-impacting symptom (or SLO burn start) to page
• False page rate: pages that don’t lead to action or confirmed impact
• Time-to-triage: page to “we know the component + suspected change”
• Rollback/pause automation rate: % of deploy regressions stopped automatically

Metrics that will lie to you:

• “Number of alerts configured”
• “Number of dashboards”
• “CPU average”
• “Total requests” without error/latency context

A simple operating model I’ve seen work across Kubernetes shops, monoliths, and messy hybrids:

1. Pick your top 3 customer journeys (checkout, login, search, etc.).
2. Define SLOs and burn-rate alerts.
3. Add 2–3 saturation/queue leading indicators per journey.
4. Make every page rollout-aware.
5. Gate canaries on those same metrics.

Do that, and you’ll usually see MTTD drop in weeks—not quarters.

“If your incident detection doesn’t know about your deploys, you’re just waiting to be surprised.”

When you’re ready to stop guessing

If you’re drowning in alerts, shipping AI-generated code faster than your on-call can reason about it, or you’ve got a Kubernetes cluster that only one person “understands,” GitPlumbers is the team you call when you want signal, not theater. We’ll help you define the leading indicators, wire them into triage, and make your rollouts self-protecting—without a six-month observability replatform.

• See how we approach Reliability & Observability: https://gitplumbers.com/services/reliability-observability
• Read a real code rescue engagement: https://gitplumbers.com/case-studies/production-fire-drill-to-slo

If you want, share one recent incident timeline and your current alert list—we’ll tell you (bluntly) which 20% to keep and how to automate the rest.