The Correlation Engine: Predicting Incidents and Rolling Back Before Users Notice

The 2 a.m. outage you could’ve seen coming

I’ve watched teams chase ghosts at 2 a.m. because CPU looked fine while users were timing out. Turned out a “trivial” config change flipped a retry policy, thrashed a downstream, and p99 climbed slowly for 20 minutes before the hard fail. No one correlated the symptom to the cause because their dashboards were full of vanity metrics and had zero change context.

The fix wasn’t another dashboard. It was a correlation engine: treat every change (deploys, flags, configs, schema migrations) as first-class telemetry, wire it to metrics/logs/traces, and then plug that into rollout automation. When p99 slope ticks up after a canary, you auto-pause before Twitter notices.

GitPlumbers has built this pattern at fintechs, gaming, and SaaS with Prometheus/Grafana/Loki/Tempo, OpenTelemetry, and Argo Rollouts/Flagger. Here’s the playbook we’ve seen actually work.

Measure what predicts pain, not what flatters you

If your top panels are avg CPU and request count, you’re optimising for vibes. The indicators that predict incidents are about saturation, stability, and error budgets.

• Error budget burn rate: multi-window alerts catch fast and slow burns.
• Tail latency slope (p95/p99): the slope is the canary-in-the-coal-mine.
• Queue growth: growing faster than drain rate = inevitable brownout.
• Resource throttling: cgroup CPU throttling, JVM GC pause spikes.
• Backpressure signals: gRPC UNAVAILABLE, Kafka consumer lag divergence.

PromQL examples you can ship today:

# 30m error budget burn rate (adjust SLO target and windows)
(sum(rate(http_request_errors_total{job="api",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m])))

# p99 now vs 10 minutes ago; alert if >20% worse
histogram_quantile(0.99, rate(http_server_duration_seconds_bucket{job="api"}[5m]))
  > on(job)
1.2 * histogram_quantile(0.99, rate(http_server_duration_seconds_bucket{job="api"}[5m])) offset 10m

# Queue growth rate (RabbitMQ as example)
deriv(rabbitmq_queue_messages_ready{queue="checkout"}[10m]) > 100

# CPU throttling ratio > 20%
rate(container_cpu_cfs_throttled_seconds_total{container!=""}[5m])
  /
rate(container_cpu_cfs_periods_total{container!=""}[5m]) > 0.2

# JVM GC pause time spike
rate(jvm_gc_pause_seconds_sum{job="payments"}[5m]) > 0.5

And ship multi-window, multi-burn-rate SLO alerts (Google SRE model):

# prometheus-rule.yaml
groups:
  - name: slo-burn
    rules:
      - alert: APIErrorBudgetBurnFast
        expr: |
          (sum(rate(http_request_errors_total{job="api",code=~"5.."}[1m]))
           / sum(rate(http_requests_total{job="api"}[1m]))) > (0.01 * 14) # 1% SLO, 14x for 5m
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "API fast burn"
      - alert: APIErrorBudgetBurnSlow
        expr: |
          (sum(rate(http_request_errors_total{job="api",code=~"5.."}[5m]))
           / sum(rate(http_requests_total{job="api"}[5m]))) > (0.01 * 6)  # 6x for 1h
        for: 1h
        labels: {severity: ticket}
        annotations:
          summary: "API slow burn"

Correlate symptoms with change events by default

Everything is a change: deploys (ArgoCD), flags (LaunchDarkly), infra (Terraform), schema migrations (Flyway), even traffic shifts (Istio/Linkerd). Put these in your telemetry path.

• Annotate Grafana on every rollout/flag flip.
• Enrich traces with deployment, commit_sha, feature_flag.
• Emit change counters as metrics so Prometheus can join by labels.
• Log the trace_id and use Tempo + Loki to jump between them.

OTel Collector to enrich spans/logs with change context:

# otel-collector.yaml
receivers:
  otlp:
    protocols: {grpc: {}, http: {}}
processors:
  attributes:
    actions:
      - key: deployment
        action: upsert
        value: ${DEPLOYMENT}
      - key: commit_sha
        action: upsert
        value: ${GIT_COMMIT}
      - key: feature_flag
        action: insert
        value: ${FEATURE_FLAG}
exporters:
  otlp:
    endpoint: tempo:4317
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
service:
  pipelines:
    traces: {receivers: [otlp], processors: [attributes], exporters: [otlp]}
    logs:   {receivers: [otlp], processors: [attributes], exporters: [loki]}

Annotate Grafana from CI/CD (ArgoCD, GitHub Actions):

curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H 'Content-Type: application/json' \
  -X POST "$GRAFANA_URL/api/annotations" \
  -d "{\"tags\":[\"deploy\",\"api\"],\"text\":\"deploy api ${GIT_SHA}\",\"time\":$(date +%s%3N)}"

Emit simple change metrics from your deploy job:

echo "deployment_started_total{service=\"api\",commit=\"$GIT_SHA\"} 1" | curl -s --data-binary @- http://pushgateway:9091/metrics/job/deploy

Queries that actually answer “what changed?”

Grafana gets you 80% of the way if you model changes as metrics/annotations.

• Before/after delta tied to a deploy:

# p99 delta within 15m window of last deploy event (via offset)
with deploy as (max_over_time(deployment_started_total{service="api"}[15m]))
  histogram_quantile(0.99, rate(http_server_duration_seconds_bucket{job="api"}[5m]))
  - histogram_quantile(0.99, rate(http_server_duration_seconds_bucket{job="api"}[5m])) offset 15m

• Correlate error logs to traces and owners:

# Loki: extract trace and service, count ERRORs by service
{app="api"} |= "ERROR" | json | line_format "{{.trace_id}} {{.service}}"

Then pivot in Tempo to see the exact spans post-deploy. If you’re on Grafana, set a derived field so trace_id in logs is clickable into Tempo.

• Detect backpressure early:

# Consumer lag growing faster than production rate (Kafka example)
rate(kafka_consumergroup_lag{group="payments"}[5m])
  > on(topic) group_left
1.2 * rate(kafka_topic_partition_current_offset[5m])

• Throttle detection tied to binpack mistakes:

(rate(container_cpu_cfs_throttled_seconds_total{namespace="prod",pod=~"api-.*"}[5m])
 /
 rate(container_cpu_cfs_periods_total{namespace="prod",pod=~"api-.*"}[5m]))
  > bool 0.2

Pro tip: keep queries readable and fast. I’ve seen teams ship PhD-grade PromQL that melts the TSDB during incidents. Prefer pre-aggregated metrics and sane label cardinality.

Close the loop: automate triage and rollouts

Correlation is only useful if it drives action. Let your rollout controller consume the same metrics.

Argo Rollouts AnalysisTemplate that gates a canary on burn rate and p99 slope:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: api-canary-analysis
spec:
  metrics:
    - name: burn-rate-fast
      interval: 1m
      count: 5
      successCondition: result < 0.14 # 14x budget over 5m is bad
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            (sum(rate(http_request_errors_total{job="api",code=~"5.."}[1m]))
             / sum(rate(http_requests_total{job="api"}[1m])))
    - name: p99-regression
      interval: 1m
      count: 10
      successCondition: result < 1.2
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            histogram_quantile(0.99, rate(http_server_duration_seconds_bucket{job="api"}[5m]))
            /
            histogram_quantile(0.99, rate(http_server_duration_seconds_bucket{job="api"}[5m])) offset 10m

Flagger gives you a similar pattern for canaries on Istio/Linkerd with out-of-the-box Prometheus checks. The net effect: if a change correlates with leading indicators going red, the rollout pauses or rolls back automatically. Your MTTR trends down because the worst incidents never fully land.

For triage, route alerts to the right owners using your service catalog (Backstage) and change context:

# alertmanager.yml (snippet)
route:
  receiver: default
  routes:
    - matchers:
        - service=~"payments|checkout"
      receiver: team-payments
      continue: true
receivers:
  - name: team-payments
    slack_configs:
      - channel: "#team-payments"
        title: "{{ .CommonLabels.service }} incident ({{ .Status }})"
        text: "Triggered by {{ .CommonLabels.commit }} deploy; runbook: https://backstage/catalog/{{ .CommonLabels.service }}"

The 30-day plan that actually sticks

I don’t care how pretty your architecture diagram is; if you can’t land this in a month, it will be deprioritized to death.

1. Instrument the spine
   • Propagate trace_id through gateways and jobs; log it.
   • Deploy Tempo + Loki + Prometheus + Grafana (or your vendor equivalents). Terraform it.
2. Add change context
   • Emit deploy/flag/migration events as metrics and Grafana annotations from CI/CD (ArgoCD/GitOps or Jenkins/GitHub Actions).
   • Enrich spans via OTel Collector with deployment, commit_sha.
3. Define SLOs and leading indicators
   • Two SLOs per critical service. Ship multi-window burn alerts.
   • Add queue growth, throttling, GC, and p99 slope panels.
4. Gate rollouts
   • Wire Argo Rollouts/Flagger AnalysisTemplates to your Prom queries.
   • Start with canary on 5% traffic; auto-pause on regression.
5. Triage automation
   • Route Alertmanager using ownership metadata (Backstage or labels).
   • Add “what changed?” panels and Grafana links from alerts.
6. Prove it
   • Run a chaos drill: break a dependency, watch canary auto-pause, confirm MTTR and pages reduced. Document the runbook.

Pitfalls I’ve seen (and how to dodge them)

• Cardinality explosions: dumping commit_sha as a label on high-cardinality series will nuke Prometheus. Put change IDs on annotations and traces; keep metrics labels tame.
• Vibe coding: AI-generated dashboards or copy-pasted PromQL with no ownership rots fast. Treat queries as code, review them, and test with promtool.
• Collector roulette: multiple sidecars fighting over traces/logs. Standardize on a single OTel Collector per node or namespace.
• Tool tourism: six APMs and no shared mental model. Pick a stack and make it the default. I’ve shipped Grafana OSS and Datadog with equal success—consistency is what matters.
• False positives: tie alerts to SLOs and multi-window rates. You’re not monitoring the system; you’re protecting user experience.

“If you can’t answer ‘what changed?’ in 30 seconds, you’re doing incident response on hard mode.”

Results we’ve seen when this sticks

• 30–60% reduction in MTTR within two quarters.
• 40% fewer user-visible incidents because bad canaries never fully roll out.
• Faster on-call: median “time-to-change-identification” drops from minutes to seconds.
• Exec trust climbs because postmortems show data-backed causality, not vibes.

If your stack has legacy bits or AI-generated glue code that’s quietly dropping trace_id or mislabeling metrics, we’ve done the vibe code cleanup and AI code refactoring to make the telemetry trustworthy.

Where GitPlumbers fits

We parachute in to wire the correlation engine, clean up your telemetry, and bolt it to your rollout controllers. We’ve untangled Istio/Linkerd mesh metrics, fixed broken OTel setups, replaced bespoke “observability middleware” that interns wrote during a hack week, and shipped SLOs that engineers actually believe in. If you want a partner that will ship something real in 30 days and not leave behind a binder of buzzwords, we should talk.