Test Design & Avoiding Confounds Questions

Learn common experiment pitfalls: time-of-week biases (weekend vs. weekday users behave differently), seasonal effects (holiday periods skew conversion), learning effects (users adapt to new features over time), and network effects (one user's action influences another). Practice identifying these confounds in scenarios and designing tests to avoid them. Understand random assignment and why it matters.

MediumTechnical

161 practiced

You want to test a new 'share' button on a social platform where acceptance by one user can cause friends to adopt features (spillover). Design an experiment to measure both direct treatment effects and indirect spillover. Discuss randomization unit (user vs cluster), metrics to capture spillover, identification strategy, and handling of incomplete network visibility.

MediumTechnical

100 practiced

You evaluate 20 metrics across 5 segments after an experiment and find multiple p < 0.05 results. How would you control the false positive rate across metrics and segments? Compare Bonferroni correction, Benjamini-Hochberg FDR, and hierarchical/multilevel modeling approaches, and give guidance on which to use in different business contexts.

MediumTechnical

88 practiced

When and why would you use ANCOVA/regression adjustment in A/B test analysis? Describe the assumptions it relies on, the potential power gain, and the situations where adjusting for covariates (especially post-randomization covariates) could introduce bias.

MediumTechnical

73 practiced

Explain the pitfalls of optional stopping (checking results repeatedly and stopping when p < 0.05) and describe statistical techniques that allow interim looks without inflating Type I error (alpha-spending functions, Pocock/O'Brien-Fleming, and Bayesian sequential approaches). Recommend practical rules your organization should adopt.

EasyTechnical

98 practiced

You're testing a new sign-up flow and stakeholders debate using 'time-to-first-action' vs 'sign-up rate' as the primary metric. As the data scientist, pick a primary metric and justify your choice. Also list 2-3 guardrail metrics you would track and why they matter for avoiding confounded decisions.

Unlock Full Question Bank

Get access to hundreds of Test Design & Avoiding Confounds interview questions and detailed answers.

Join thousands of developers preparing for their dream job.