Do we need a production-scale environment to get meaningful results?

You need production-like behavior more than identical scale. Use realistic data cardinality, warm caches to expected hit rates, and simulate third-party latency/failures. For read-heavy paths, run small, controlled tests in production with rate limits. For write-heavy paths, use service virtualization and shadow traffic to avoid data corruption.

Open vs. closed workload models—why should I care?

Closed models tie request rate to response time, masking tail latency (coordinated omission). Open models send requests at a fixed arrival rate regardless of response. Real users are open load. Use tools like k6’s constant-arrival-rate, wrk2, or Vegeta to generate open load and expose backpressure and queueing effects.

How do I include third parties without risking billable calls?

Virtualize them with WireMock/Mountebank/Toxiproxy. Record real responses, inject jitter/timeouts/429s, and rate-limit outbound calls. Validate that your timeouts, retries, and circuit breakers behave under load. Then run a small canary against the real provider off-peak to confirm assumptions.

What metrics should gate a release?

User-facing ones aligned to SLOs: p95/p99 latency per journey, error rate, and a business proxy (e.g., checkout success). Support with saturation (DB connections, thread pool queue depth) for root cause. Bake thresholds into CI so regressions block merges automatically.

How often should we re-run load tests?

At minimum before major promos and quarterly to refresh capacity headroom. Mature teams run weekly soaks and trigger targeted load tests on significant infra or dependency changes (database version upgrades, feature flags that change data access patterns).

Performance-optimization · Oct 3, 2025 · 10 minute read

The Load Test That Caught a $3M Outage Before Marketing Did

Stop chasing CPU graphs. Validate user-facing behavior under real stress, tie it to revenue, and ship with a margin of safety.

Back to all posts

The Load Test That Caught a $3M Outage Before Marketing Did

Key takeaways

Implementation checklist