From Pager Hell to Predictable On-Call: How SLOs Cut Pages 65% in 90 Days

The incident treadmill we walked into

I walked into a Monday standup where the on-call had slept on the couch in the office. Again. The company (let’s call them MarketForge) runs a high-traffic marketplace on AWS EKS with Istio, Prometheus, and Alertmanager. They'd bolted on Datadog for host metrics, Sentry for errors, and PagerDuty for paging.

The problem: incidents were defined by whatever metric tripped first. CPU spikes? Page. Pod restarts? Page. A minor 5xx blip on a non-critical endpoint at 3 a.m.? Page. They were doing drive-by dashboarding and vibe debugging. AI-assisted PRs were shipping faster than the monitoring could keep up, and the team was drowning in noise.

• Average pages per week: 38 (with bursts >60 during releases)
• MTTR: ~6 hours
• MTTD: ~20 minutes (read: humans noticed before dashboards)
• Change failure rate: ~32%
• Compliance constraint: PCI scope on checkout; zero tolerance for silent failures there

I’ve seen this movie: without SLOs, you’re optimizing for graphs, not users. We flipped the script.

Why SLOs changed the game

Dashboards are for humans; SLOs are contracts with your users. When we centered incident response on SLOs, three things happened immediately:

• We stopped paging on infra noise and started paging on user pain.
• We could quantify risk with error budgets instead of arguing about severity.
• We created a common language across engineering, product, and compliance.

We anchored on two simple rules:

1. Define SLIs around critical user journeys: login, search, checkout.
2. Alert only when SLO error budgets burned at meaningful rates (fast and slow).

We didn’t fix incidents; we fixed what we alert on.

What we implemented (the boring, critical details)

We kept it boring on purpose. Fancy observability with no governance is just expensive noise.

1. SLIs for critical paths

• Availability: ratio of 2xx|3xx to all responses per service
• Latency: p95 under a threshold (e.g., 300ms for search, 500ms for checkout)
• Source: Istio metrics exported to Prometheus

PromQL examples:

# Checkout availability (5xx + throttles considered errors)
sum(rate(istio_requests_total{destination_workload="checkout", response_code=~"5..|429"}[5m]))
/
sum(rate(istio_requests_total{destination_workload="checkout"}[5m]))

# Checkout latency SLI (p95 under 500ms)
histogram_quantile(0.95,
  sum by (le) (rate(istio_request_duration_milliseconds_bucket{destination_workload="checkout"}[5m]))
) < 0.5

2. SLOs with error budgets

• Checkout availability SLO: 99.9% over 28 days
• Search latency SLO: p95 < 300ms for 99% of requests over 28 days
• Login availability SLO: 99.95% over 28 days

We codified these with Sloth so SLOs, alerts, and dashboards are generated from one YAML source of truth and deployed with ArgoCD.

# sloth.yaml
service: checkout
slos:
  - name: http-availability
    objective: 99.9
    description: "HTTP 2xx/3xx ratio over 28d"
    labels:
      team: core-commerce
      tier: critical
    sli:
      events:
        error_query: |
          sum(rate(istio_requests_total{destination_workload="checkout",response_code=~"5..|429"}[5m]))
        total_query: |
          sum(rate(istio_requests_total{destination_workload="checkout"}[5m]))
    alerting:
      name: checkout-slo
      labels:
        severity: page
      annotations:
        runbook: https://runbooks.marketforge.internal/checkout/slo
      page_alert: { disable: false }
      ticket_alert: { disable: false }

3. Multi-window, multi-burn rate alerts

We used the Google SRE pattern to catch both fast burns (explosions) and slow burns (leaks):

• Fast burn: 2h window, burn rate > 14x (page now)
• Slow burn: 6h and 24h windows, burn rate > 6x and > 1x (ticket or page depending on tier)

# prometheus-rules.yaml (generated by Sloth, simplified)
groups:
- name: slo-burn
  rules:
  - alert: SLOErrorBudgetBurnFast
    expr: |
      (
        sum(rate(istio_requests_total{destination_workload="checkout",response_code=~"5..|429"}[5m]))
        /
        sum(rate(istio_requests_total{destination_workload="checkout"}[5m]))
      ) > (14 * (1 - 0.999))
    for: 10m
    labels:
      severity: page
      service: checkout
    annotations:
      summary: "Checkout SLO fast burn"
      runbook: "https://runbooks.marketforge.internal/checkout/slo"
  - alert: SLOErrorBudgetBurnSlow
    expr: |
      (
        avg_over_time(
          (
            sum(rate(istio_requests_total{destination_workload="checkout",response_code=~"5..|429"}[5m]))
            /
            sum(rate(istio_requests_total{destination_workload="checkout"}[5m]))
          )[6h:]
        ) > (6 * (1 - 0.999))
      )
    for: 30m
    labels:
      severity: ticket
      service: checkout
    annotations:
      summary: "Checkout SLO slow burn"
      runbook: "https://runbooks.marketforge.internal/checkout/slo"

4. Pager routing that respects sleep

Alertmanager routed only severity: page from SLO burns to PagerDuty. Everything else opened a ticket in Jira or Slack.

# alertmanager.yaml (routing snippet)
route:
  receiver: default
  routes:
  - matchers:
    - severity="page"
    receiver: pagerduty
    group_by: [service]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 3h
  - matchers:
    - severity="ticket"
    receiver: jira
receivers:
- name: pagerduty
  pagerduty_configs:
  - routing_key: ${PAGERDUTY_KEY}
- name: jira
  webhook_configs:
  - url: https://jira.marketforge.internal/hooks/alerts

5. GitOps everything

• SLO YAML lived under sre/slos/ in the mono-repo.
• ArgoCD synced PrometheusRule CRDs, Alertmanager config, and Grafana dashboards.
• Changes required a PR, code review, and a canary rollout. No click-ops.

6. Make it visible

We carved out a Grafana folder: one dashboard per service, top-left panel is remaining error budget over 28 days, plus burn rate sparkline. The first graph every on-call saw was user impact, not pod counts.

How incident response actually changed

We didn’t just add alerts; we rewrote the on-call contract.

• Paging policy: Only SLO burn pages. Node/pod/K8s noise became tickets with rational priorities.
• Triage flow started at the SLI panel, not the pod list. The second thing on-call checked was the ArgoCD deploy history. 80% of SLO burns correlated with a deploy in the last 30 minutes. Shocking, I know.
• Rollback and flag strategy: Argo Rollouts for canary, LaunchDarkly for feature flags. If a canary burned 2% of the error budget in 15 minutes, rollout abort was the default, not a debate.
• Runbooks: Each SLO had a runbook with kubectl one-liners, istioctl proxy-status, SLI queries, and feature flag kill switches.

# Quick triage snippets from the runbook
kubectl -n checkout get pods -o wide --sort-by=.status.containerStatuses[0].restartCount

# Check last deploy
argocd app history checkout | head -n 5

# Compare SLI before/after deploy
promtool query instant http://prometheus:9090 \
  'sum(rate(istio_requests_total{destination_workload="checkout",response_code=~"5..|429"}[5m]))
   /
   sum(rate(istio_requests_total{destination_workload="checkout"}[5m]))'

The social change was the hardest part. We had to deprogram the “CPU>85% == page” reflex. But once engineers saw reduced noise and clearer priorities, they leaned in.

Results after 30/60/90 days

We measured hard outcomes, not vibes.

• Pages/week: 38 → 14 (−63%) by day 60; stabilized at 13–16 by day 90
• MTTR: 6h → 1h 50m at day 30 → 48m at day 90 (−87%)
• MTTD: 20m → 5m (multi-window alerts caught slow burns early)
• Change failure rate: 32% → 12% (canaries + flag kills on burn)
• On-call satisfaction (internal survey): 2.1/5 → 4.0/5
• Unplanned downtime on checkout (28d): 220 minutes → 24 minutes
• Compliance posture: PCI evidence packs included SLO dashboards and error budget policy; zero NCs in the audit

Business impact wasn’t subtle: conversion recovered 1.8 points after we stopped drowning checkout with noisy restarts and started defending the SLO. Product stopped arguing with SRE about “is this critical?” We had numbers.

Lessons learned (and what I’d do differently)

• Don’t start with 20 SLOs. We started with three services and two SLIs each. That was enough to flip the culture.
• Pick SLO targets you can actually meet. Shipping a 99.99% SLO on day one just means permanent pages and no credibility.
• Keep SLIs boring. We resisted “weighted blended” weirdness. Ratios and histograms won.
• Codify or it didn’t happen. YAML + Sloth + ArgoCD meant changes were reviewable and auditable.
• Tie to release policy. Error budget exhaustion paused non-critical launches for a sprint. Product grumbled, then loved the predictability.
• Watch for AI-induced regressions. The fastest route to a burned budget was an “optimizing” AI patch that subtly changed retry semantics. SLOs turned those from Friday-night mysteries into Tuesday-morning blips.

Do this next week: a 7-step playbook

1. List your top three user journeys and pick an availability and latency SLI for each.
2. Set SLOs with a 28d window and error budget policies (what pauses when you breach?).
3. Generate Prometheus rules via Sloth (or hand-roll if you must) and deploy with ArgoCD.
4. Implement two alert rules per SLO: fast (2h burn) pages, slow (6h/24h) tickets.
5. Route non-SLO alerts away from PagerDuty. Sleep is a feature.
6. Write runbooks that start with SLI graphs and recent deploys. Practice once a month.
7. Add canary/flags to shorten MTTR: Argo Rollouts + LaunchDarkly is a strong combo.

If you want a sanity check on your first SLOs or help wrangling the PromQL, this is exactly what we do at GitPlumbers. We’ve cleaned up enough AI-generated “observability” YAML to know where the footguns are.