We don’t have good telemetry yet. Where do we start without boiling the ocean?

Start with the four signals that pay rent fast: error ratio (for SLO), p95 latency, backlog length (Kafka/SQS/Sidekiq), and CPU throttling. Add simple Prometheus rules and one KEDA/HPA scaler off backlog. Ship via GitOps so it sticks. Expand to pool saturation and concurrency next.

Should we use ML/forecasting for capacity?

Use ML for business forecasting (expected traffic) if you have seasonality; for incident prediction, slopes + headroom + burn rate consistently outperform fancy models in real-time. We’ve seen Prophet/Holt-Winters be fine for staffing and scheduling; production protection still hinges on leading indicators and automations.

How do we keep costs under control while autoscaling more aggressively?

Define a utilization band (e.g., 45–65%), add scale-in policies (cooldowns, smaller step-downs), and alert on under-utilization. Prefer bin-packing nodes with the right pod requests/limits and use cluster-autoscaler with spot capacity for bursty workers.

What about multi-tenant platforms (shared DBs, shared Kafka)?

Model bottlenecks per tenant where possible (labels by `tenant`), but keep recording rules at the service level to avoid cardinality blowups. Apply quotas (pool sizes, partitions per tenant), and autoscale workers on the aggregate backlog while alerting on top-N tenants driving saturation.

We’re not on Kubernetes. Does this still apply?

Yes. Use ASGs scaling on SQS depth, Kafka lag (CloudWatch or Prometheus `cloudwatch_exporter`), and gate deploys in Spinnaker or GitHub Actions with PromQL checks. The model (leading indicators -> automation) is infra-agnostic.

Reliability-observability · Oct 20, 2025 · 10 minute read

Predictive Capacity Planning That Doesn’t Lie: Leading Indicators, Not Vanity Dashboards

Build models that call the surge 30 minutes early and auto‑gate your rollouts. No more praying to CPU%.

Back to all posts

Predictive Capacity Planning That Doesn’t Lie: Leading Indicators, Not Vanity Dashboards

Key takeaways

Implementation checklist