Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

EasyTechnical

57 practiced

Explain conceptually how baseline conversion rate, minimum detectable effect (MDE), alpha, and power determine sample size. Give a numeric example: baseline conversion=5%, MDE=0.5% absolute lift, alpha=0.05, power=0.8 — explain the steps you'd take to compute sample size (no code required).

MediumTechnical

58 practiced

Compare Bayesian A/B testing to frequentist approaches: discuss interpretation of results, handling of optional stopping, prior selection, computation, and practical trade-offs for product decision timelines.

EasyTechnical

76 practiced

Describe multiple comparison problems when running experiments with many variants or metrics. Explain Bonferroni correction, Holm-Bonferroni, and Benjamini-Hochberg (FDR). For a business team, when would you recommend controlling FDR versus family-wise error rate?

HardTechnical

49 practiced

You ran an experiment that failed to show lift. Describe a structured post-mortem process: data validation checks, power and MDE review, instrumentation audit, exploratory HTE analysis, business-context review, and how to decide whether to iterate, pivot, or stop work on the idea.

EasyTechnical

81 practiced

When designing an experiment, how do you decide the randomization unit (user, session, cookie, device, account)? Describe consequences of choosing an inappropriate unit, and how you would validate that randomization is functioning correctly in production.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.