InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

MediumTechnical
65 practiced
You're interested in long-term LTV impact but the product team needs a quick signal. Describe an experiment design and analysis plan that captures both short-term conversion and signals predictive of long-term LTV, accounting for novelty effects and delayed outcomes.
MediumTechnical
58 practiced
Design a staged rollout plan for a mobile app feature given low daily traffic. Propose a ramp schedule (for example 1%, 5%, 20%, 100%), list burn-in checks at each stage, explain how to adjust for statistical power at each step, and specify rollback criteria.
MediumTechnical
72 practiced
Baseline conversion is 10%. You want to detect an absolute uplift of 1 percentage point (10% -> 11%) with 80% power and alpha=0.05 in a two-sided test with 1:1 allocation. Describe the sample size calculation steps and any assumptions. How does required sample size change if you run a 3-variant test?
HardTechnical
59 practiced
An experiment shows a statistically significant positive aggregate effect, but a high-value customer segment shows a negative effect. Describe a decision framework to decide whether to scale, pause, or run follow-ups. Include economic impact calculation, segmentation weighting, and proposed next experiments.
MediumTechnical
50 practiced
You're testing a pricing page redesign for a freemium SaaS product. Which metric would you pick as the primary metric (options: trial-signup-rate, trial-to-paid conversion, revenue-per-user) and why? List trade-offs and at least two guardrail metrics you would monitor.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.