Do I need a feature flag vendor to do this right?

No, but it helps. The critical piece is deterministic, server-side assignment and explicit exposure logging. You can implement that with LaunchDarkly/Optimizely, or with a small in-house service. What matters: a seeded hash, consistent unit IDs, and durable logging.

How do I handle bots and crawlers in experiments?

Filter them at ingestion (user-agent lists, behavior heuristics) and maintain an allowlist for analysis. Don’t rely only on UA strings—add rate and path-pattern heuristics. Exclude flagged traffic from SRM and metrics by default, but report the excluded proportion as a guardrail.

What about sequential peeking?

If you peek, you need sequential tests (e.g., SPRT, alpha spending). Alternatively, fix your horizon (e.g., 7 days) and don’t peek. Whatever you choose, encode it in your pipeline configs so analysts can’t accidentally change the rules mid-flight.

We have cross-device users. Can we still trust results?

Yes, if your identity graph is good enough and consistent with assignment. Prefer account-level randomization when feasible. If not, constrain the experiment to contexts where ID is stable (e.g., post-login flows) or accept dilution and quantify it in pre-experiment power analysis.

How do I backfill without breaking decisions already made?

Version your metrics. Write new results to a new table/version with a migration note and comparison report. Don’t rewrite history in-place. If the backfill changes conclusions, escalate with a formal experiment audit—it’s a data incident, treat it like one.

Data-engineering · Oct 30, 2025 · 10 minute read

Stop Shipping Fake Wins: The A/B Pipeline That Doesn’t Lie

Design A/B testing data pipelines that produce trustworthy results and business value, not ghost wins.

Back to all posts

Stop Shipping Fake Wins: The A/B Pipeline That Doesn’t Lie

Key takeaways

Implementation checklist