Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

HardTechnical

0 practiced

Hard: Implement a regression-based analysis (in Python/pandas or pseudocode) that estimates treatment effect with interaction terms for segments (e.g., country, device), includes robust standard errors clustered by user, and produces a table of segment-level treatment effects with CIs. Describe assumptions and limitations.

HardTechnical

0 practiced

Compare sequential testing under a frequentist alpha-spending approach with a fully Bayesian decision framework for early stopping. Discuss how each controls error rates, how priors affect outcomes, and practical considerations for automated stopping in high-velocity growth experiments.

MediumTechnical

0 practiced

Case study: An experiment shows a statistically significant +1.2% lift in click-through rate (p=0.02) but a negative guardrail: -0.8% in 7-day retention (p=0.12). The PM wants to ship. Walk through your analysis plan to recommend rollout or rollback. Include additional checks, power considerations for the retention metric, and possible mitigations if you proceed.

EasyTechnical

0 practiced

Easy: For a payment checkout optimization experiment, suggest 5 quick sanity checks you would run within the first week of data collection to decide whether the experiment is healthy enough to continue gathering data. Explain why each check matters.

HardSystem Design

0 practiced

Design a deterministic online assignment service (API) for user-level experiment bucketing that supports versioning, experiment overrides, and replayability. Provide API endpoints, input/output formats, hashing/versioning scheme, and how you'd test correctness at scale. Include how to manage migrations when updating bucketing logic.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.