Won’t automatic rollbacks hide real issues?

No—they surface issues faster. The rollback buys you time and contains blast radius. You still create an incident, attach the trigger evidence, and fix forward. The point is to trade a brief blip for a long-running outage.

How do we avoid flapping rollbacks?

Use short but non-zero windows (1–5 minutes), relative comparisons to stable, and a small failure limit (1–2). Add a cool-down before reattempting. Record why the rollback triggered to tune thresholds over time.

What about stateful services and database migrations?

Use expand/contract patterns. Ship the new schema first, write code compatible with both, then cut over via a feature flag. Rollbacks revert app behavior, not destructive schema changes. For data backfills, treat them as jobs with their own kill switches.

Does this work outside Kubernetes?

Yes. Spinnaker + Kayenta can do canary analysis for VMs. AWS CodeDeploy supports blue/green with CloudWatch alarms. The design principles—SLO-based triggers, short windows, and Git-managed configs—are the same.

Who owns the templates and thresholds?

Platform/SRE owns the shared templates and tooling; service teams own their SLOs and thresholds. Changes land via PR. That split keeps consistency without stealing domain expertise.

Release-engineering · Nov 5, 2025 · 10 minute read

Your Canary Isn’t a Seatbelt: Automated Rollbacks That Cut MTTR, Not Corners

Ship fast, sleep at night. Wire rollbacks to real SLOs and stop arguing in Slack during incidents.

Back to all posts

Your Canary Isn’t a Seatbelt: Automated Rollbacks That Cut MTTR, Not Corners

Key takeaways

Implementation checklist