InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

HardTechnical
0 practiced
An experiment is underpowered and fails to reach significance. List statistical and product-level strategies to salvage insights without committing p-hacking: meta-analysis across similar experiments, pooling, Bayesian updating, and pre-specified subgroup analysis. For each method explain assumptions and how to avoid misleading conclusions.
MediumTechnical
0 practiced
You are planning a checkout optimization experiment. List guardrail metrics (at least four) you would include and propose concrete alert thresholds for each to automatically pause or rollback the experiment.
MediumTechnical
0 practiced
Baseline conversion is 10%. You want to detect an absolute uplift of 1 percentage point (10% -> 11%) with 80% power and alpha=0.05 in a two-sided test with 1:1 allocation. Describe the sample size calculation steps and any assumptions. How does required sample size change if you run a 3-variant test?
EasyTechnical
0 practiced
You want to test a personalized recommendation algorithm. Describe how you would select target segments and control groups to ensure valid inference and appropriate generalizability. Explain sampling strategy, inclusion/exclusion criteria, and how to handle new users with scarce history.
EasyTechnical
0 practiced
When and why should a PM pre-register an experiment? List the essential fields you would include in a pre-registration template (for example: hypothesis, primary metric, guardrails, sample size, stopping rules, analysis plan). Provide a short example of a filled template for an onboarding test.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.