How do we keep pre-merge under 10 minutes if our suite is huge?

Shard tests across more runners, adopt test impact analysis (Bazel, Jest changedSince, pytest-testmon), and move heavyweight integration/E2E to pre-release. Cache dependencies and test artifacts aggressively. If you’re still slow, your builds aren’t hermetic or you’re doing work that belongs post-merge.

What if we don’t control upstream provider services for contract tests?

Use provider states and a Pact Broker. Even if a third party won’t run provider verifications, you can validate your consumer contracts against the latest published provider versions and gate your deploys on what you can control. For external APIs, add synthetic smoke against a sandbox or a mock that replays real traces.

Is canary overkill for small teams?

Not if you use managed rollouts. Argo Rollouts or Flagger need a few YAML stanzas and Prometheus. The cost is small; the payoff in lower CFR and faster recovery is huge. Start with a simple 10% step and a single SLO-aligned metric.

Release-engineering · Oct 24, 2025 · 10 minute read

Green Builds, Red Incidents: The Automated Test Gate That Actually Catches Regressions

If your change failure rate is creeping up while CI stays green, your test gates aren’t telling the truth. Here’s the automation we deploy to cut CFR, shrink lead time, and make recovery boring.

Back to all posts

Green Builds, Red Incidents: The Automated Test Gate That Actually Catches Regressions

Key takeaways

Implementation checklist