Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

HardTechnical

0 practiced

As an experimentation lead, propose an organizational governance model to scale experimentation: include test planning and pre-approval, experiment registry, standardized metric definitions (metric catalog), quality gates, training/onboarding, and a lightweight review board. Explain trade-offs and how you'd measure adoption and quality.

HardTechnical

0 practiced

Discuss cross-device and identity challenges for experiments: differences between cookie-based, device-based, and authenticated user IDs; risks of inconsistent assignment across devices; and how to implement deterministic assignment and stitching while minimizing bias.

MediumTechnical

0 practiced

Implement a Python function `required_sample_size_prop(p0, d, alpha=0.05, power=0.8, two_sided=True)` that returns the required sample size per arm for a two-sample test of proportions using the normal approximation. Document any assumptions and edge cases your function handles.

EasyTechnical

0 practiced

Explain Type I error, Type II error, statistical power, and significance level in the context of product A/B testing. Use concrete product-oriented examples (e.g., conversion lift vs. false alarms) and explain the trade-offs between reducing alpha and maintaining power.

HardTechnical

0 practiced

After an experiment shows a strong conversion uplift but also a large increase in refunds and customer complaints, craft a rigorous decision framework for rollout vs partial rollout vs rollback. Include immediate monitoring, segmentation to isolate issues, staged rollout strategies, and stakeholder communication plan.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.