A/B Test Design & Statistical Rigor Questions

Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.

HardTechnical
45 practiced

You are evaluating a price increase (for example, raising a marketplace take rate or introducing a new fee) in a two-sided marketplace with network effects between buyers and sellers. Design an experiment that accounts for spillovers between the two sides: specify the randomization scheme, including whether to randomize by buyer, seller, or a shared cluster, how you would detect and quantify cross-side externalities, and what analysis approach you would use to estimate the long-run revenue impact under these network effects.

MediumTechnical
48 practiced

Explain CUPED (Controlled Experiments Using Pre-Experiment Data) as a variance-reduction technique for A/B tests. Describe what pre-experiment data it requires, the assumptions it relies on, and in plain terms how the adjustment is computed. What makes a pre-experiment covariate a good or a poor choice for CUPED, and what goes wrong if you pick a poor one?

MediumTechnical
44 practiced

Beyond the initial launch experiment, why would you keep a long-run holdout group even after a feature or a pricing algorithm change has fully shipped? Explain how you would decide the size of the holdout, how long to maintain it, and what you are trying to learn from it that the original launch experiment could not tell you. How would you communicate the cost of maintaining a holdout to stakeholders who want the new experience rolled out to everyone?

HardTechnical
47 practiced

Your A/B test shows no overall lift, but a particular user segment, say mobile users, shows a statistically significant positive uplift. How would you validate whether this is a genuine heterogeneous treatment effect rather than a false positive from looking at many segments? What analyses would you run, and if you're not yet certain, what decision process would you use to decide whether to ship for that segment, run a confirmatory follow-up experiment, or abandon the finding?

MediumTechnical
51 practiced

You need to plan how long an experiment must run. Given daily unique visitors, the traffic allocation per variant, baseline conversion rate, desired minimum detectable effect, alpha, and power, show how to compute the required sample size per variant and then convert that into an expected number of days to run the test. State the assumptions and rounding choices you make along the way.

Unlock Full Question Bank

Get access to all 22 A/B Test Design & Statistical Rigor interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.