InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

MediumTechnical
0 practiced
Given tables: users(user_id, signup_date), events(user_id, event_type, event_date), and assignments(user_id, variant), write SQL to compute weekly active users (WAU) per variant and a retention curve (7- and 30-day retention) for users exposed in the first week. Describe assumptions about exposure and attribution.
HardTechnical
0 practiced
Implement in Python (or describe the algorithm) a sequential hypothesis test using an alpha-spending approach. Inputs: total_alpha, schedule_of_interim_looks (list of cumulative information proportions), observed z-scores at each look. Output: whether to stop for efficacy, futility, or continue. Describe how you would choose a spending function (e.g., Pocock vs O'Brien-Fleming).
MediumTechnical
0 practiced
You're given: baseline conversion = 3.0%, daily eligible users = 200,000, desired power = 80%, alpha = 0.05, MDE (relative) = 10%. Calculate required sample size per variant and the expected test duration if traffic is evenly split and conversion measurement takes 7 days after exposure. Show your calculations and assumptions.
EasyTechnical
0 practiced
Define three criteria that distinguish a testable hypothesis from a guess in product experimentation. Provide a one-sentence example of a guess and convert it into a testable hypothesis using the required structure. Explain why your converted hypothesis is testable.
HardTechnical
0 practiced
You receive experiment logs with non-random missing data, high attrition, and partial noncompliance (some users not exposed to assigned variant). Outline an end-to-end analysis plan including ITT (intention-to-treat), CACE (complier average causal effect) estimation, weighting strategies, and robustness checks you would run. State assumptions required for each method.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.