What if we can’t log prompts due to compliance?

Hash prompts with per-tenant salts to dedupe for caching/metrics without storing raw text. Use on-by-default redaction in OpenTelemetry and store raw prompts only with explicit tenant consent.

We’re on `pgvector` and can’t introduce another DB. Is stabilization still possible?

Yes. Upgrade the extension, add HNSW/IVFFLAT indexes, reduce top-k, and move embedding writes to a queue. You can buy 2–3x latency improvement without leaving Postgres.

Do we have to switch LLM providers to fix cost/latency?

Usually no. Multi-model fallbacks and right-sizing context length get you most of the win. We often keep Azure OpenAI for regulated tenants and add a cheaper model for non-critical paths.

How long does this kind of stabilization take?

We aim for 2–4 weeks. Week 1 is triage (circuit breakers, caching, caps). Weeks 2–3 tackle data paths and observability. Week 4 is polish and handoff with dashboards and runbooks.

Case-studies · Oct 3, 2025 · 10 minute read

The AI Copilot That Melted at P95: Stabilized Under Real Customer Load in 21 Days

A Series C SaaS shipped an AI sidecar that cratered under real users. We cut P95 from 2.8s to 650ms, slashed token spend 64%, and stopped the pager from ruining weekends.

Alex Ramirez

Principal Engineer, GitPlumbers

20 years scaling brittle systems into boring ones. Ex-Netflix SRE, ex-Stripe platform reliability. I turn flaky AI features into margin-friendly products.

“We went from apologizing to upselling. The AI features stopped being a liability.”

Back to all posts

The AI Copilot That Melted at P95: Stabilized Under Real Customer Load in 21 Days

Key takeaways

Implementation checklist