InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

MediumTechnical
0 practiced
When selecting the target segment for an experiment limited to logged-in users across multiple geographies, what inclusion and exclusion criteria would you set? Discuss trade-offs between statistical power, safety, and generalizability, and how to implement segmentation in the BI pipeline.
MediumTechnical
0 practiced
For a recommendation algorithm experiment, propose a primary metric and at least three guardrail metrics that would ensure the test does not produce harmful side effects. Explain how each metric maps to business goals and how to compute them in your BI reports.
EasyTechnical
0 practiced
Compare A/B testing, multivariate testing, and staged rollouts. For each method: give a one-sentence definition, one strength, one weakness, and a short example of when a BI analyst should recommend it.
MediumTechnical
0 practiced
Given a table metrics(date, user_id, variant, metric_value), outline a SQL or pandas approach to estimate treatment effect using difference-in-differences for a rollout where treatment started on a single date. Include key assumptions and how you would check them in your BI tools.
HardTechnical
0 practiced
An experiment shows a non-significant 2% drop in a secondary metric that could jeopardize long-term business health. Create a follow-up plan: pre-specified analyses, power calculations for confirmatory testing, possible secondary experiments, and rollback criteria. Include how to quantify risk and decision thresholds.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.