InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

HardSystem Design
72 practiced
Instrumentation drift caused by logging changes has introduced a small but persistent bias in a key metric across experiments. Describe methods to detect instrumentation drift automatically, how to backfill or reconcile historical data, how to run validation experiments to quantify bias, and what instrumentation best practices you would implement to prevent future drift.
MediumTechnical
66 practiced
You must test a new pricing message but it must be randomized at city level because implementation differs by local teams. Baseline conversion is 5% and you estimate intra-class correlation (ICC) of 0.02. Explain how cluster randomization affects required sample size, show the design effect formula, and outline steps to compute the number of clusters required per arm, making reasonable assumptions about cluster size.
EasyTechnical
54 practiced
Describe a simple rule of thumb to estimate experiment duration given baseline conversion, required sample size, and daily eligible traffic. Then explain at least three practical factors that might lengthen the experiment beyond the calculated minimum duration and how you would account for them in planning.
HardTechnical
88 practiced
Your organization runs many concurrent experiments across different product areas. Propose a statistical framework and practical engineering controls to allow overlapping experiments with minimal confounding. Include strategies such as blocking, stratified randomization, regression adjustment for main effects, experiment overlap rules, and tooling for discovering and flagging risky overlaps.
MediumTechnical
69 practiced
Describe a quantitative prioritization framework to rank experiment ideas using estimated impact, confidence, and development cost. Define how you would estimate each component (for example, expected uplift, probability of success, and engineering days), how to combine them into a single score, and provide a short example calculation comparing two competing experiments.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.