InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

HardTechnical
0 practiced
Discuss the trade-offs between optimizing for a short-term KPI (e.g., click-through rate) and long-term user health (e.g., retention). Propose an objective function or multi-metric decision rule that balances these and includes fairness or business constraints.
MediumTechnical
0 practiced
List SQL and pragmatic checks you would run to validate experiment instrumentation and cohort assignment before declaring an experiment healthy. Include example queries or pseudo-SQL to detect unequal assignment, duplicate assignments, and missing event rates.
MediumTechnical
0 practiced
You observe a statistically significant 3% lift in sign-ups but a 6% drop in weekly retention (statistically significant). Draft a decision rule on whether to roll out the variant, run follow-up experiments, or abandon it. Include primary/secondary metrics, thresholds, and immediate investigative steps you would take using BI reports.
EasyTechnical
0 practiced
You see a 12% drop in weekly active users on a core workflow. Propose three distinct, testable hypotheses (phrased in if-change-then-because) that could explain the drop. For each, list one data source or dashboard you'd query to validate whether the hypothesis is plausible before running experiments.
EasyTechnical
0 practiced
Explain randomization and bucketing strategies for web and mobile experiments. Compare deterministic hashing, server-side assignment, and client-side assignment; list two pitfalls for each approach and how you would detect them using BI queries.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.