Do I still need a circuit breaker if the provider SDK has retries and timeouts?

Yes. Provider SDKs protect their edge, not your SLOs. You need application-level timeouts and breakers tuned to your budgets, plus mesh-level protections to stop cascades within your cluster.

What’s a sane timeout for inline user flows?

Work backward from your SLO. If the request has 600ms total, give the AI hop ~250–300ms including one retry. Anything slower must fall back or be async.

How do I detect hallucinations automatically?

Use schema validation to catch structural issues, retrieval-grounding checks (does the answer cite retrieved docs?), and heuristic classifiers for toxicity/PII. Track a “validation_fail” metric and fall back or re-prompt when it trips.

What’s the fastest fallback that still looks good?

Cached last-known-good content with a short TTL, or a deterministic rules-based template. Users prefer slightly stale but correct over slow or wrong.

How do I prevent cost blowouts?

Enforce token quotas per request/user/tenant, cap context size, and monitor `ai_cost_tokens_total`. Add a cost breaker that downgrades models or refuses requests when budgets are exhausted.

Ai-delivery · Nov 2, 2025 · 10 minute read

Circuit Breakers and Fallbacks for AI: The Guardrails That Save You When Models Misbehave

LLMs will fail in production—hallucinations, drift, latency spikes, provider outages. Here’s how we wire breakers, fallbacks, and observability so those failures don’t take your business down.

Back to all posts

Circuit Breakers and Fallbacks for AI: The Guardrails That Save You When Models Misbehave

Key takeaways

Implementation checklist