InterviewStack.io LogoInterviewStack.io

Hypothesis and Test Planning Questions

End to end practice of generating clear testable hypotheses and designing experiments to validate them. Candidates should be able to structure hypotheses using if change then expected outcome because reasoning ground hypotheses in data or qualitative research and distinguish hypotheses from guesses. They should translate hypotheses into experimental variants and choose the appropriate experiment type such as A and B tests multivariate designs or staged rollouts. Core skills include defining primary and guardrail metrics that map to business goals selecting target segments and control groups calculating sample size and duration driven by statistical power and minimum detectable effect and specifying analysis plans and stopping rules. Candidates should be able to pre register plans where appropriate estimate implementation effort and expected impact specify decision rules for scaling or abandoning variants and describe iteration and follow up analyses while avoiding common pitfalls such as peeking and selection bias.

MediumSystem Design
0 practiced
Design an instrumentation QA checklist and real-time monitoring pipeline for experiments. Include pre-launch checks (unit tests, SRM quick-check), in-flight monitors (SRM, event latency, funnel drops), alerting thresholds, and post-test validation steps. Explain how you'd surface actionable issues to engineers and PMs.
MediumTechnical
0 practiced
Implement a Python function sample_size_two_proportions(baseline_rate, mde_relative, alpha=0.05, power=0.8) that returns the required sample size per variant for a two-sided two-sample z-test using the normal approximation. Assume mde_relative is a relative uplift (e.g., 0.1 for 10%). Explain formulas and assumptions in comments. Use numpy/scipy if needed.
EasyTechnical
0 practiced
Explain Type I and Type II errors in the context of A/B testing. Give one concrete business example of each (for example, shipping a harmful change because of a false positive). Discuss how choosing alpha (e.g., 0.05 vs 0.01) affects the balance between these errors and the associated business trade-offs.
MediumTechnical
0 practiced
Given these simplified schemas: users(user_id UUID PRIMARY KEY, assigned_variant TEXT, assigned_at TIMESTAMP) and events(event_id UUID, user_id UUID, event_type TEXT, occurred_at TIMESTAMP), write a Postgres SQL query to compute daily conversion rate per variant over a 14-day experiment window (conversion defined as event_type='purchase' within 14 days of assigned_at). Include counts, conversion_rate, and 95% Wilson confidence intervals per day and variant.
HardTechnical
0 practiced
Provide pseudocode or a Python outline that estimates ITT and CACE (complier average causal effect) using instrumental variables when some users assigned to treatment did not receive it. Assume arrays assigned (0/1), exposed (0/1, actual exposure), and outcome (numeric). Include formula and implementation steps and explain assumptions.

Unlock Full Question Bank

Get access to hundreds of Hypothesis and Test Planning interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.