InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

EasyTechnical
56 practiced
In the Applied Scientist role, explain what makes a hypothesis testable versus a guess. Using the 'If X then Y because Z' format, write a clear, testable hypothesis for improving homepage click-through rate where X is a homepage change and Y is the measurable outcome. Explicitly identify the treatment, the expected metric change, the rationale grounded in user behavior or prior data, and how you would measure success in a real experiment.
MediumTechnical
60 practiced
Draft the essential components of a pre-registered analysis plan for an A/B test that is expected to affect revenue per user. The plan should include the primary hypothesis, exact metric definitions (denominator, numerator, attribution window), inclusion/exclusion criteria, treatment assignment logic, analysis window, handling of outliers and winsorization, approach to multiple metrics, and a stopping rule that preserves Type I error.
HardTechnical
71 practiced
Revenue per user is heavy-tailed with many zeros, making analytic sample size formulas unreliable. Outline and write pseudocode for a simulation-based power calculation that models the revenue distribution, applies a proposed treatment effect (for example a 5% multiplicative uplift on non-zero revenue), and estimates power. State how you would validate the simulation inputs and report uncertainty in the estimated power.
HardTechnical
63 practiced
You want to build an uplift model to personalize offers. Design the randomized experiment required to train and validate the uplift model: specify treatment and control splits, how to allocate samples between training and holdout sets, labeling approach for uplift, evaluation metrics for uplift accuracy and business value, and safeguards to avoid bias during model training and evaluation.
MediumTechnical
73 practiced
A product experiment requires users to opt in to a new flow, creating potential selection bias. Explain how inverse probability weighting (IPW) could be used to adjust treatment effect estimates, list the assumptions required for IPW to be valid, and propose alternative experimental designs that avoid opt-in selection bias.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.