What if our legacy service isn’t on Kubernetes?

No problem. The same approach works on VMs. Use node exporters, run Prometheus as a service, ship logs via Fluent Bit to Loki/Elastic, and expose an HTTP metrics endpoint. You can still attach the OpenTelemetry Java agent and point to a collector over TCP. GitOps can be achieved with Ansible or Terraform plus a CI runner.

How do we pick the right latency threshold for SLOs?

Start with user experience and current performance. Look at p95 from your 2-week baseline and talk to product about what’s acceptable. If p95 is 420ms today and users notice slowness beyond ~500ms, set an initial SLO of 99% under 500ms, then tighten once breakers/timeouts are in.

Won’t tracing be too expensive?

Not if you sample intelligently. Start with 1–5%, use tail-based sampling on anomalies if your collector supports it, and drop high-cardinality attributes. Retain spans for shorter periods (24–72h) and metrics longer (30–90d).

What’s the fastest way to get SLOs into Git?

Use Sloth. Define availability/latency SLOs in YAML, `sloth generate` to produce Prometheus rules and alerts, and let ArgoCD/Flux roll them out. Add dashboards as code so panels don’t drift.

How do we align SLOs with business impact?

Run a monthly error budget review with product. If you’re burning budget, pause feature work to harden. If you’re consistently under budget, you can accelerate. Tie these decisions to real incidents and user-facing metrics (conversion, drop-offs).

Guides · Nov 21, 2025 · 9 minute read

The Legacy Service That Finally Stopped Paging Us: Progressive Observability + SLOs That Stick

You don’t harden a legacy service with a Big Bang overhaul. You level it up in weeks, not quarters—instrument, baseline, define SLOs, then enforce error budgets. Here’s the playbook that’s actually worked for us in production.

Back to all posts

The Legacy Service That Finally Stopped Paging Us: Progressive Observability + SLOs That Stick

Key takeaways

Implementation checklist