Do I need Kubernetes for this?

No. The same approach works on VMs. Use node_exporter for system metrics, blackbox_exporter for HTTP checks, Prometheus static scrape configs, and the OpenTelemetry Collector as a systemd service. The SLO math doesn’t care where it runs.

What if my legacy app can’t be instrumented easily?

Start outside-in: black-box checks and NGINX or Envoy access logs give you RED metrics. Add a sidecar for metrics if possible. Trace only around the edges (DB proxy, HTTP client libraries) before touching app code.

How do I pick SLO targets without historical data?

Start conservative: 99.5% availability and a latency threshold users feel (e.g., 500ms p95) over 28d. After 2–4 weeks, adjust based on observed budgets and incident impact. Document the rationale so finance and product buy in.

Will tracing blow up my costs?

Not if you sample. Head sample 1–5% and tail-sample slow/error traces. Keep spans minimal and short-lived. Store only a few days of traces hot; metrics should drive most decisions.

Can I keep Datadog/New Relic and still do this?

Yes. You can still codify SLOs with Sloth and query Datadog metrics, or use their SLO objects. The progressive approach and burn-rate alerting pattern are tool-agnostic.

Guides · Nov 8, 2025 · 10 minute read

The Legacy Service That Stopped Paging at 2 a.m.: Progressive Observability and SLOs That Stick

You don’t need a platform rewrite to sleep through the night. You need a measured path to metrics, logs, traces, and SLOs that drive decisions.

Back to all posts

The Legacy Service That Stopped Paging at 2 a.m.: Progressive Observability and SLOs That Stick

Key takeaways

Implementation checklist