Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

MediumTechnical

0 practiced

Write a SQL query or outline SQL steps that compute Sample Ratio Mismatch (SRM) for an experiment with variants A and B. Input table: exposures(user_id, experiment_id, variant). Output should include counts per variant, expected counts assuming equal allocation, chi-square statistic and a p-value approximation or logic for flagging SRM if p < 0.01.

MediumTechnical

0 practiced

Define a primary metric and a guardrail metric in the context of an experiment. Provide two concrete examples of primary and guardrail metrics for a checkout redesign experiment and explain how you would prioritize trade-offs if the primary metric improves but a guardrail metric shows small degradation.

MediumTechnical

0 practiced

How do you prioritize experiments across multiple product initiatives when resources are limited? Describe one or two prioritization frameworks (e.g., ICE, PIE), explain how you'd adapt them for long-term metrics or technical debt experiments, and describe how you'd incorporate expected learnings and risk into prioritization.

MediumTechnical

0 practiced

Explain Sample Ratio Mismatch (SRM): what it is, common causes (e.g., instrumentation bugs, segmented allocation, filtering post-randomization), how to detect it statistically in the experiments pipeline, and short-term and long-term remediation strategies you would take as a data analyst.

MediumTechnical

0 practiced

You need to run an experiment comparing two checkout flows but also want to understand feature interactions with a loyalty badge that only some users have. Describe an experimental design that allows you to estimate the main effect and the interaction with the loyalty badge, and outline the analysis model you would run (specify variables and interaction term).

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.