How do we retrofit exposure-first design if our app only emits outcome events?

Stand up an assignment service (GrowthBook SDKs or a tiny Go/Python service) that deterministically buckets with a logged salt and emits exposure events synchronously at decision time. Backfill only to support historical dashboards—never to calculate current experiment results.

Is CUPED always worth it?

If your metric is noisy (think revenue per user, time on site), CUPED typically reduces variance 10–30%. Compute the pre-period feature once daily and join. Be explicit in reporting (flags/notes) so PMs understand adjusted vs. raw.

Where should metric definitions live: dbt, BI, or the experiment tool?

dbt. The experiment tool consumes pre-aggregated tables; BI builds on the same models. We’ve seen BI-defined metrics drift. Keep SQL in Git with tests and versioning.

How do we detect identity churn or bot traffic biasing results?

Add diagnostics: user_id alias churn rate, exposure-to-outcome lag distribution, bot score distributions by variant. Alert on anomalies and exclude upstream via a shared filter model referenced by all metrics.

Can we do this on Redshift or Databricks?

Yes. Replace Snowflake/BigQuery with your warehouse, keep the same patterns: contracts, deterministic bucketing, dbt models, and quality gates. We’ve shipped this on Redshift RA3 and on Databricks SQL with Delta tables.

Data-engineering · Nov 15, 2025 · 9 minute read

The A/B Pipeline That Lied To Us (And How We Stopped Shipping Fake Wins)

If your A/B results swing wildly or arrive late, your pipeline—not your product—is the problem. Here’s the battle-tested design GitPlumbers uses to make experiment data boringly reliable.

Back to all posts

The A/B Pipeline That Lied To Us (And How We Stopped Shipping Fake Wins)

Key takeaways

Implementation checklist