Where should I put the circuit breaker: mesh or app?

Both. Use the mesh (Istio/Linkerd/Envoy) to shed load and eject bad upstreams. Use an app-level breaker to trip on semantic failures (schema invalid, low grounding) the mesh can’t see.

How do I detect hallucinations automatically?

Use a retrieval-overlap gate for RAG, JSON schema validation for structured outputs, and a lightweight verifier (NLI/entailment) for critical answers. Log and sample for human review.

What’s a sane fallback order?

Smaller/faster model; compact prompt; retrieval-only with citations; cached answer; then human handoff. Don’t invent data in fallbacks.

How do I pick SLOs for AI endpoints?

Start with user experience: p95 under 1–1.5s for interactive flows, <2% error including schema failures, and <1% grounding failures on sampled evals. Adjust based on margin of error and business tolerance.

Ai-delivery · Oct 25, 2025 · 9 minute read

Circuit Breakers for LLMs: How We Stop Hallucinations, Drift, and Latency Spikes From Taking Production Down

Your AI chain is only as strong as its weakest upstream. Here’s how we keep LLM-driven systems from faceplanting—and what to instrument so you actually know when they’re failing.

Back to all posts

Circuit Breakers for LLMs: How We Stop Hallucinations, Drift, and Latency Spikes From Taking Production Down

Key takeaways

Implementation checklist