Statistical Foundations for Experimentation Questions

Core statistical concepts and inference needed to design analyze and interpret experiments. Topics include hypothesis testing p values confidence intervals Type One and Type Two errors the relationship between sample size variability and interval width statistical power minimum detectable effect and effect size versus practical significance. Candidates should be able to choose and explain common statistical tests such as t tests and chi square tests contrast Bayesian and frequentist approaches at a conceptual level and describe variance estimation and variance reduction techniques. The topic covers corrections for multiple comparisons sequential testing and the risks of peeking and p hacking common misconceptions about p values and limitations of inference such as confounding and selection bias. Candidates should also be able to translate statistical findings into clear language for non technical stakeholders and explain uncertainty and limitations.

HardTechnical

68 practiced

Optional stopping invalidates naive p-values. Describe the Sequential Probability Ratio Test (SPRT) and martingale-based always-valid p-values as formal solutions to optional stopping. Explain assumptions underlying each approach, how to choose stopping boundaries, and how to estimate long-run Type I error under plausible model misspecification.

MediumTechnical

58 practiced

You run an A/B test measuring revenue per user with high variance. Describe three variance-reduction techniques (blocking/stratification, covariate adjustment including CUPED, and outcome transformation). For each technique explain assumptions, implementation steps, expected effect on variance, and potential pitfalls (e.g., conditioning on post-treatment variables).

MediumTechnical

86 practiced

Discuss the theory and practice of adjusting for baseline covariates in randomized experiments. Explain why covariate adjustment does not introduce bias when randomization is valid, how it typically affects variance and precision, and where to be careful (for example, adjusting for post-treatment variables or misspecifying functional form). Include brief notes on implementation (regression vs stratification) and inference.

HardTechnical

48 practiced

Design an experiment and analysis plan to measure uplift on a rare event (for example, purchase rate 0.1% per visit). Discuss sample-size implications, choice of the metric (absolute vs relative uplift), statistical models suitable for rare counts (Poisson, negative binomial, zero-inflated), power-enhancement strategies (longer windows, grouping, surrogate metrics), and how you would present and report uncertainty.

MediumTechnical

62 practiced

Design a permutation (randomization) test to compare two groups on a skewed metric (for example, number of messages sent). Specify the algorithmic steps, the null hypothesis, the choice of test statistic, how to compute a p-value, and discuss computational optimizations for large datasets (e.g., approximate Monte Carlo, stratified permutations). Also describe when exact permutation is infeasible and how to handle that.

Unlock Full Question Bank

Get access to hundreds of Statistical Foundations for Experimentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.