Do we need a managed feature store or is Feast enough?

If you already run Redis/Kafka/S3 and have platform/SRE coverage, Feast plus a small amount of glue code is plenty for a first production system. If your team is lean or compliance is heavy (PII controls, fine-grained RBAC, HIPAA/PCI), managed platforms like Tecton or Hopsworks reduce blast radius and speed up audits.

How do we prevent training-serving skew in practice?

Define features once in code and use the exact same transformation logic for historical and online materialization. Enforce point-in-time joins in training, set explicit defaults/TTLs, and block PRs that introduce alternate code paths. Add a CI check that compares feature schema and versions between training and serving artifacts.

What’s the minimum viable observability for AI in prod?

Prometheus metrics for feature freshness, null ratios, and p95/p99 latencies; model counters with version/feature-hash labels; OpenTelemetry traces across retrieval → model → policy; and drift metrics (PSI/KL) on top features. Everything else is a nice-to-have until this is green.

How do we guard against LLM hallucinations without killing UX?

Constrain outputs with strict schemas, require citations tied to retrieved context, set confidence thresholds, and implement fast fallbacks (search/templates). Users prefer a reliable, occasionally conservative response over confident nonsense. Log rejections and refine prompts/contexts with those cases.

What SLOs make sense for feature stores?

Start with: online lookup p95 < 25ms, freshness SLO per feature (e.g., 95% of reads < 2 minutes old), null/default ratio < 1% for critical features, materialization lag < 5 minutes for streaming features. Tie business SLOs (approval/answer rates) to deploy gates in canaries.

Ai-delivery · Nov 5, 2025 · 10 minute read

Feature Stores That Don’t Gaslight You: Serving the Same Truth Online and Offline

If your model returns different answers depending on the path that fed it features, you don’t have an AI system — you have improv. Here’s the feature store architecture and guardrails that actually hold up in prod.

Back to all posts

Feature Stores That Don’t Gaslight You: Serving the Same Truth Online and Offline

Key takeaways

Implementation checklist