How do we start without boiling the ocean?

Pick one critical service with a clear SLO, instrument it end-to-end with OpenTelemetry, define three leading indicators (e.g., retry rate, queue lag, CPU throttling), and gate its canary with an AnalysisTemplate. Backtest the alerts against the last five incidents. Expand from there.

Can we do this with Datadog/New Relic instead of Prometheus/OTel?

Yes. The principles are the same: propagate trace context, model a service graph, capture change events, define leading indicators, and gate rollouts. Datadog Monitors with APM traces and Deployment Tracking, or New Relic NerdGraph + AIOps, can implement the same loop.

Isn’t correlation just fancy alert spam?

It becomes spam if you don’t rank suspects, include change events, or tie it to action. Correlation should reduce pages by finding predictive signals and either auto-rolling back or handing you the top two suspects with links to runbooks and traces.

What about machine learning for anomaly detection?

Great once your basics are rock solid. We’ve seen simple cross-correlation + service graph + burn rate beat “AI” anomaly boxes until the data hygiene, sampling, and labeling mature. Then add ML for seasonality and multi-variate drift.

Reliability-observability · Oct 11, 2025 · 10 minute read

The Correlation Engine That Saved Our Canary (And Your Weekend)

Stop paging on vanity graphs. Design correlation that predicts incidents, shortens triage, and drives automated rollbacks.

Riley Hart

Principal Reliability Engineer, GitPlumbers

20 years building and fixing distributed systems from bare metal to service meshes. Ex-Netflix platform, ex-Shopify production engineering. I ship guardrails, not dashboards.

Dashboards are for humans. Correlation is for machines that don’t get tired.

Back to all posts

The Correlation Engine That Saved Our Canary (And Your Weekend)

Key takeaways

Implementation checklist