Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

HardTechnical

0 practiced

Design an online experiment to evaluate a personalized recommender that continuously retrains on streaming user feedback. Address issues: bias from online learning (training on treatment-influenced data), cold-start, normalization of scores across variants, interleaving vs full A/B, and valid inference for long-running adaptive systems.

MediumTechnical

0 practiced

Describe methods to estimate heterogeneous treatment effects across segments: regression with interaction terms, stratified analysis, tree-based causal methods (e.g., causal forests), and uplift trees. Discuss trade-offs in interpretability, variance, and multiple-testing.

MediumTechnical

0 practiced

Implement a Python function `required_sample_size_prop(p0, d, alpha=0.05, power=0.8, two_sided=True)` that returns the required sample size per arm for a two-sample test of proportions using the normal approximation. Document any assumptions and edge cases your function handles.

HardSystem Design

0 practiced

Design a production-ready multi-armed bandit system to optimize click-through across variants where traffic patterns are non-stationary. Discuss algorithm choices (epsilon-greedy, UCB, Thompson sampling), exploration schedule, burning-in period, offline evaluation, and how to integrate with existing A/B testing workflows.

MediumTechnical

0 practiced

Compare Bayesian A/B testing to frequentist approaches: discuss interpretation of results, handling of optional stopping, prior selection, computation, and practical trade-offs for product decision timelines.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.