Do I still need retries if I have a circuit breaker?

Yes, but keep retries bounded by a budget and backoff with jitter. The breaker protects you during sustained failure; retries handle transient blips. Never retry on timeouts forever or you’ll amplify load.

What’s a good first fallback for customer-facing answers?

A retrieval-only answer from your documentation with a confidence watermark. It’s fast, safe, and often good enough. Follow with a smaller/cheaper model or a templated response if retrieval confidence is low.

How do I detect hallucinations automatically?

Use a combination of schema checks, retrieval confidence thresholds, and offline evaluations over golden datasets. Track an eval failure rate metric and alert on spikes. True hallucination detection is probabilistic—design for mitigation, not perfect detection.

Should I run multiple LLM providers?

If uptime or price volatility matter, yes. Use mesh-level routing with outlier detection and feature-flag switches. Keep prompts/templates provider-agnostic and versioned so you can shift traffic without a redeploy.

Where do breakers live—in app code or the mesh?

Both. App-level breakers handle business-aware fallbacks and timeouts. Mesh-level policies protect the fleet and let you eject bad upstreams quickly. They’re complementary, not substitutes.

Ai-delivery · Nov 12, 2025 · 10 minute read

The Circuit Breaker That Saved Our LLM: Fallbacks, Guardrails, and Observability That Actually Work

Your LLM won’t take down prod if you treat it like the flaky dependency it is. Breakers, fallbacks, and hard telemetry turned our worst AI failure modes into manageable blips.

Back to all posts

The Circuit Breaker That Saved Our LLM: Fallbacks, Guardrails, and Observability That Actually Work

Key takeaways

Implementation checklist