Do we need a managed feature store or is Feast enough?

If your org already runs Snowflake/Kafka/Redis well and you have platform engineers, Feast is enough to start. If you’re short on platform capacity or need enterprise governance on day one, Tecton or Hopsworks will save you time (and incidents). We’ve deployed both; the key is agreeing on contracts and observability first.

How do we detect online/offline skew before it hurts revenue?

Compute the same feature set on a sampled window via both paths, log values with a shared entity key and timestamp, and compare distributions nightly with Evidently. Alert only on sustained divergence that correlates with KPI deltas. Shadow testing before canarying is your friend.

What’s a reasonable SLO for feature fetch latency?

For most web-facing inference paths, p99 < 50ms for feature fetch is a good starting point. Budget 20–30% of your end-to-end SLO to features, the rest to model inference and post-processing. Track and alert on p90 and p99 separately.

How do we stop LLM hallucinations in customer-facing flows?

Enforce strict input schemas, require minimum retrieval confidence (or citations count) before answering, add content moderation, and implement abstention with clear UX. Log every decision with trace IDs and sample full transcripts for offline review.

Ai-delivery · Oct 7, 2025 · 10 minute read

Stop Training on One World and Serving Another: A Feature Store Architecture That Holds Up in Prod

Q: How do we stop LLM hallucinations in customer-facing flows?

Enforce strict input schemas, require minimum retrieval confidence (or citations count) before answering, add content moderation, and implement abstention with clear UX. Log every decision with trace IDs and sample full transcripts for offline review.

Implement feature store architectures that keep online and offline features consistent, with real instrumentation, observability, and safety guardrails across AI-enabled flows.

Alex Moran

Partner, GitPlumbers

20 years building and rescuing distributed systems at scale. Ex-Twitter SRE, ex-Shopify platform, helped teams ship ML and LLM systems that don’t page at 3 a.m.

Features are your API to the model—if that contract lies in prod, the model will, too.

Back to all posts

Stop Training on One World and Serving Another: A Feature Store Architecture That Holds Up in Prod

Key takeaways

Implementation checklist