Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

MediumTechnical

59 practiced

Estimate the required sample size per variant for a two-arm experiment comparing proportions. Inputs: baseline conversion p0 = 0.05, desired power = 0.8, alpha = 0.05 (two-sided), and MDE = 0.01 absolute (i.e., from 5% to 6%). Explain the formula and show the calculation steps conceptually. State assumptions and how sample size changes with MDE and alpha.

MediumTechnical

41 practiced

Given the following tables:

exposures(exposure_id, user_id, variant, exposure_time)
purchases(purchase_id, user_id, amount, purchase_time)

Write an ANSI SQL query that computes per-variant the 7-day revenue per exposed-user, counting only the first exposure per user and summing purchases within 7 days after that first exposure. Include nulls for users with no purchases so the denominator is total exposed users.

HardTechnical

48 practiced

A pattern emerges where several short-term experiments show small positive lifts on immediate conversion but subsequent retention analysis shows negative impact on 30-day retention. Prepare an investigation plan: what follow-up analyses you would run, cohorts and time windows to inspect, alternative explanations to rule out, and potential mitigation experiments.

HardTechnical

44 practiced

Write a Python function compute_power_two_proportions(p0, p1, n_a, n_b, alpha=0.05) that estimates statistical power for a two-proportion test using normal approximation (allow unequal n). Return estimated power. Comment assumptions and limitations in docstring. No external libraries required but pseudocode is acceptable.

MediumTechnical

45 practiced

Given a list of pre-experiment covariates, write SQL to compute standardized mean differences (SMD) for each covariate between treatment and control groups to check balance. Input: users(user_id, age int, country varchar, signup_ts), exposures(user_id, variant). Output: covariate, smd_value. Explain interpretation thresholds.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.