InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

MediumTechnical
0 practiced
You're given: baseline conversion = 3.0%, daily eligible users = 200,000, desired power = 80%, alpha = 0.05, MDE (relative) = 10%. Calculate required sample size per variant and the expected test duration if traffic is evenly split and conversion measurement takes 7 days after exposure. Show your calculations and assumptions.
EasyTechnical
0 practiced
What is pre-registration in the context of experiments? List the key elements a data analyst should include in a pre-registration document (at minimum 6 items) for a high-impact product test, and explain why preregistration reduces bias.
MediumTechnical
0 practiced
You are the data analyst asked to prioritize a backlog of 12 proposed experiments. Describe a concise scoring rubric that includes expected impact, ease of implementation, required sample size/time, and learn/no-learn value. Show how you would compute a priority score and how you would present this to product leadership.
HardTechnical
0 practiced
Compare frequentist and Bayesian approaches for product experimentation. Discuss differences in interpretation, stopping flexibility, required sample planning, and how each approach affects decision-making and stakeholder communication for growth experiments.
MediumTechnical
0 practiced
Discuss the implications of running experiments across user segments (e.g., new users vs. returning users). How would you pre-specify subgroup analyses, what sample size adjustments are needed, and how do you guard against over-interpreting noisy subgroup results?

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.