How do I know if I should shard my database or add read replicas first?

Add read replicas if reads dominate and you can route read-only traffic safely; monitor replica lag. If write p95 is high, or a few tenants/keys are hot, partition by a stable key (tenant, region) and keep writes local. Sharding is a topology choice—don’t do it until you’ve fixed N+1s, indexing, and connection pooling.

Can I rely on CPU-based HPA for stateless services?

Not for user latency SLOs. CPU is a lagging, noisy signal. Use concurrency (in-flight requests), request rate, and p95 latency as HPA inputs. For consumers, scale on queue depth or lag via KEDA.

What’s the minimal setup to avoid cache stampedes?

Enable per-key locks or request coalescing, use `proxy_cache_use_stale updating`/`stale-while-revalidate`, jitter TTLs, and set reasonable upstream timeouts. A small amount of coordination prevents a thundering herd during invalidations.

How do I scale AI-generated code that’s already in prod?

First, stop the bleeding: circuit breakers, rate limits, and bounded retries. Then perform a vibe code cleanup—remove N+1s, add connection pools, parameterize queries, and write SLO-driven tests. We’ve rescued several teams in under two weeks by pairing those fixes with better autoscaling signals.

What tooling do you recommend to keep infra costs in check while scaling?

Use `Prometheus`/`Grafana` for cost-aware dashboards, `Cluster Autoscaler`/`Karpenter` for bin-packing, `Terraform` to right-size instance classes, and load-based scaling to avoid idle capacity. Measure cost per 1k requests alongside p95.

Performance-optimization · Nov 17, 2025 · 10 minute read

Horizontal Scale Without Regret: Stateless vs Stateful, What Actually Works

What I learned scaling a checkout from 15 rps to 2,500 rps without blowing up p95 or the database.

Back to all posts

Horizontal Scale Without Regret: Stateless vs Stateful, What Actually Works

Key takeaways

Implementation checklist