Experimentation and Product Validation Questions

Designing and interpreting experiments and validation strategies to test product hypotheses. Includes hypothesis formulation, experimental design, sample sizing considerations, metrics selection, interpreting results and statistical uncertainty, and avoiding common pitfalls such as peeking and multiple hypothesis testing. Also covers qualitative validation methods such as interviews and pilots, and using a mix of methods to validate product ideas before scaling.

HardTechnical

64 practiced

Design a cluster-randomized experiment where the unit of randomization is the company account (cluster). Explain how intra-cluster correlation (ICC) affects power, show the formula for the design effect and how to adjust sample size, and work through a numerical example where ICC=0.05 and average cluster size is 50.

HardTechnical

67 practiced

Implement an alpha-spending schedule calculator in Python that outputs critical p-value thresholds for interim looks under Pocock and O'Brien-Fleming boundaries given a planned number of looks. The function should return thresholds per look and explain how each boundary behaves (early vs late conservatism).

HardTechnical

79 practiced

Implement Thompson Sampling for a binary conversion metric in Python for a K-armed bandit. Your implementation should simulate allocation of traffic over time, update Beta priors, and report cumulative regret and conversion rate per arm. Explain how you would evaluate this algorithm offline using historical A/B test logs.

EasyTechnical

56 practiced

Explain the difference between A/B testing (single factor) and multivariate testing (MVT). Discuss when multivariate testing is appropriate, how combinatorial explosion affects sample size and power, and a rule-of-thumb for when to choose MVT versus a sequence of A/B tests.

MediumTechnical

55 practiced

Your experiment assigns users to treatment but many assigned users never see the new feature (non-compliance). Explain intent-to-treat (ITT) analysis vs per-protocol analysis, and describe how you would estimate complier average causal effect (CACE) using instrumental variables. What assumptions are required?

Unlock Full Question Bank

Get access to hundreds of Experimentation and Product Validation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.