Experimentation Methodology and Rigor Questions

Focuses on rigorous experimental methodology and advanced testing approaches needed to produce reliable, actionable results. Topics include statistical power and minimum detectable effect trade offs, multiple hypothesis correction, sequential and interim analysis, variance reduction techniques, heterogenous treatment effects, interference and network effects, bias in online experiments, two stage or multi component testing, multivariate designs, experiment velocity versus validity trade offs, and methods to measure business impact beyond proximal metrics. Senior level discussion includes designing frameworks and practices to ensure methodological rigor across teams and examples of how to balance rapid iteration with safeguards to avoid false positives.

MediumTechnical

117 practiced

You need to test three independent features simultaneously: UI change (A), recommendation ranking (B), and incentives (C). Design a factorial experiment: specify the allocation, how to test for interactions, and how to estimate marginal effects. What sample size considerations are unique to factorial designs?

MediumTechnical

65 practiced

Describe the Sample Ratio Test (SRT). How would you implement automated SRT checks in a metrics pipeline, what thresholds would you use, and what are common causes and mitigations for a sample ratio mismatch?

HardTechnical

70 practiced

You discovered many subgroup analyses showing significant effects across dozens of demographic slices. Propose a principled approach to control false discoveries across subgroups while identifying meaningful heterogeneity. Include discussion of hierarchical modeling, FDR control, and minimum subgroup size constraints.

EasyTechnical

101 practiced

Compare family-wise error rate (FWER) and false discovery rate (FDR). In a platform running thousands of concurrent A/B tests and many metric checks, when would you use Bonferroni correction versus Benjamini-Hochberg versus hierarchical Bayesian approaches for multiplicity control?

EasyTechnical

101 practiced

List the most common sources of bias in online experiments (instrumentation, randomization failure, novelty effects, etc.). For each source, give one practical detection method you would implement in an experimentation platform.

Unlock Full Question Bank

Get access to hundreds of Experimentation Methodology and Rigor interview questions and detailed answers.

Join thousands of developers preparing for their dream job.