Batch or streaming for exposures?

Stream the exposure log with exactly-once or idempotent upserts so analysis stays fresh and deduped. Batch the heavy metrics if you must, but keep exposure truth in a low-latency stream so you can alert on drift and dedupe in near-real time.

How do we backfill without corrupting analysis?

Backfill with the same idempotent keys and write paths; tag backfilled rows and exclude them from freshness SLOs. Re-run dbt models with a controlled `valid_from` watermark and do a dual-run diff (shadow table) before swapping.

What about bots and internal traffic?

Filter at ingestion using bot lists and UA heuristics; mark internal IP ranges and authenticated staff. Keep these rules versioned and testable, and store the flags so analysts can audit exclusions.

We use a vendor (Optimizely/LaunchDarkly/Statsig). Still relevant?

Yes. Vendors help with assignment and stats, but you still own exposure truth, metric definitions, joins, and quality SLOs. We routinely integrate vendor exposure streams into the same guardrails and registry.

We have AI-generated analytics code. Is that a problem?

Only if you trust it without tests. We see LLM-written SQL that ignores idempotency and windows. Wrap it with contracts, dbt tests, and lineage, or bring in a cleanup pass (we call it vibe code cleanup) before it touches production decisions.

Data-engineering · Nov 14, 2025 · 10 minute read

The A/B Test Pipeline That Lied to Product: Designing Experiment Data You Can Trust

Your A/B tests aren’t failing because statistics are hard. They’re failing because your data pipeline is lying. Here’s the design that keeps experiment results reliable, fast, and tied to business value.

Back to all posts

The A/B Test Pipeline That Lied to Product: Designing Experiment Data You Can Trust

Key takeaways

Implementation checklist