Do I need a feature store if we’re only serving an LLM?

If you’re doing RAG or re-ranking, you already have features: embeddings, recency, click signals. Version your encoder and corpus, set freshness SLAs for the index, and track groundedness. A lightweight store (Feast + Redis) works; managed options reduce toil.

Feast vs. Tecton vs. Databricks Feature Store—what should I pick?

If you have strong platform engineering and want OSS control, Feast is solid. If you want enterprise support, lineage, and governance out of the box, Tecton is strong. If you’re all-in on Databricks (Delta/Unity Catalog), their Feature Store keeps everything in one plane. The real win is standardizing transforms and enforcing parity—tooling is second.

What SLOs should we set for features?

Common targets: p95 feature lookup under 100–150ms, freshness lag under 2–5 minutes for streaming features, null rate under 1% per feature, and error budget aligned with your customer SLO. Track cache hit rate and distribution skew alerts.

How do we avoid training/serving skew?

Single code path for transforms, point-in-time joins for training, versioned feature definitions, and pre-prod validation comparing offline vs. shadow online samples. Block deploys if skew exceeds thresholds.

How do we deploy safely without stalling the roadmap?

Shadow traffic + canary with Argo Rollouts. Automate rollback based on Prometheus metrics (p95/p99, freshness, null rate). Feature flag new features to cohorts. This adds hours, not weeks, and saves you days of incident response.

Ai-delivery · Oct 4, 2025 · 9 minute read

Stop Blaming the Model: Build a Feature Store That Doesn’t Lie in Prod

Your model is fine. Your features aren’t. Here’s the feature store architecture and guardrails we deploy when teams keep getting burned by drift, hallucinations, and latency spikes.

Alex Kim

Principal Architect, GitPlumbers

20 years shipping and fixing distributed systems at scale. Ex-Stripe, AWS, and a few startups that no longer exist. I lead GitPlumbers’ AI reliability practice.

Boring prod is good prod. Your feature store is either making that possible—or making your pager go off.

Back to all posts

Stop Blaming the Model: Build a Feature Store That Doesn’t Lie in Prod

Key takeaways

Implementation checklist