Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

EasyTechnical

42 practiced

Design an event schema for instrumenting a signup funnel for experimentation. Provide the event names and key attributes for at least: impression, click, signup-start, signup-complete, and revenue. Describe how you'd make events idempotent, include unique identifiers, and version event schemas to support future changes.

HardSystem Design

42 practiced

Design a deterministic online assignment service (API) for user-level experiment bucketing that supports versioning, experiment overrides, and replayability. Provide API endpoints, input/output formats, hashing/versioning scheme, and how you'd test correctness at scale. Include how to manage migrations when updating bucketing logic.

MediumTechnical

49 practiced

You want to reduce variance and increase experiment sensitivity using covariate adjustment. Explain ANCOVA, stratification, blocking, and regression adjustment approaches for binary outcomes in experiments. When is each appropriate and what are pitfalls to avoid?

EasyTechnical

90 practiced

When should a growth team use an observational causal inference approach (e.g., propensity-score matching, difference-in-differences) instead of running an A/B test? Provide 3 concrete scenarios and the limitations of observational methods compared to randomized experiments.

EasyTechnical

53 practiced

You ran a randomized A/B test but observe clear covariate imbalance between treatment and control on key demographic features (e.g., age, country). List steps you would take to diagnose and address this imbalance before trusting the experiment result. Include both short-term fixes and long-term process improvements.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.