What was the single highest-leverage change?

Bounding concurrency on LLM calls per pod (a bulkhead) plus hard timeouts. It immediately stopped runaway in-flight work and reduced cascading failures into Postgres and the rest of the API.

Why not just scale Kubernetes pods until it works?

Because the LLM provider had hard rate limits. Scaling pods increased parallelism, which increased 429s, which triggered retries, which made the system less stable and more expensive.

How do you cache safely in a multi-tenant AI app?

Use tenant-prefixed cache keys, avoid cross-tenant shared blobs, and include versioning (like `docSetVersion`) in keys so you can invalidate safely when content changes.

What telemetry matters for LLM-backed endpoints?

p95/p99 latency, error rate, in-flight concurrency, queue depth, cache hit rate, and LLM-specific attributes like token counts, model/provider, and rate-limit responses.

Case-studies · Jan 9, 2026 · 8 minute read

The AI Copilot That Faceplanted at 9:05 AM: How We Got It Stable Under Real Customer Load

A production case study in cleaning up AI-assisted code, fixing LLM-induced latency spikes, and turning “works in staging” into boring reliability.

Back to all posts

The AI Copilot That Faceplanted at 9:05 AM: How We Got It Stable Under Real Customer Load

Key takeaways

Implementation checklist