What thresholds should I use for my first circuit breaker?

Start conservative: request timeout 2–3s, volume threshold 20, error threshold 50%, reset 10s. Tune based on your provider’s p95 and your user experience. Don’t forget per-try timeouts if you enable retries.

How do I prevent hallucinations in a RAG system?

Validate outputs against a JSON schema, include confidence scores and citations, and short-circuit when retrieval returns low-signal context. Use evaluator suites (promptfoo/ragas) on canary traffic to catch drift before a full rollout.

Isn’t mesh-level retry enough?

No. Retries without application timeouts and a breaker will amplify latency spikes and cost. You need app-level control for validation, logging prompt versions, and triggering safe fallbacks.

How do I do provider failover safely?

Normalize the interface, keep prompts semantically equivalent across providers, and record per-provider metrics. Start with read-only shadow traffic to the backup. Only promote after passing SLOs and eval gates.

Ai-delivery · Oct 12, 2025 · 10 minute read

Stop Letting LLMs 500 Your App: Circuit Breakers, Fallbacks, and Guardrails That Actually Work

The production playbook for when your AI services spike, drift, or flat-out hallucinate—and how to keep your app alive anyway.

Alex Ramirez

Principal Engineer, GitPlumbers

20 years shipping and rescuing distributed systems at scale. Ex-Netflix platform, ex-Stripe reliability. I build guardrails so your team can ship AI without waking up PagerDuty at 3 a.m.

Boring fallbacks beat magical failures. Ship the breaker before you ship the prompt.

Back to all posts

Stop Letting LLMs 500 Your App: Circuit Breakers, Fallbacks, and Guardrails That Actually Work

Key takeaways

Implementation checklist