Experiment Design and Execution Questions

Covers end to end design and execution of experiments and A B tests, including identifying high value hypotheses, defining treatment variants and control, ensuring valid randomization, defining primary and guardrail metrics, calculating sample size and statistical power, instrumenting events, running analyses and interpreting results, and deciding on rollout or rollback. Also includes building testing infrastructure, establishing organizational best practices for experimentation, communicating learnings, and discussing both successful and failed tests and their impact on product decisions.

HardTechnical

0 practiced

You are analyzing an experiment and notice that variants were partially rolled out (assignment percentages changed mid-test) and there was some user re-assignment. Describe how you would clean and analyze the data: what assumptions you would test, which users to include in ITT, and whether a weighted analysis or time-varying treatment indicators are appropriate.

HardSystem Design

0 practiced

You're asked to explore multi-armed bandits (MAB) as an alternative to A/B testing for a personalization feature. Describe the MAB variants (epsilon-greedy, Thompson Sampling, UCB), trade-offs compared to fixed-sample A/B tests, business constraints that might prevent adoption, and how you would evaluate a bandit rollout safely.

HardTechnical

0 practiced

You're tasked with building an experimentation best-practices program for a company that currently lacks one. Draft a prioritized 90-day plan that includes training, templates (experiment brief, hypothesis template), tooling suggestions, an experiment review board, and KPIs to measure success of the program rollout.

MediumTechnical

0 practiced

Describe a robust event instrumentation scheme to support A/B testing: what core events and attributes you would require (exposure event, assignment id, variant name, user_id, timestamps, client/server), how to ensure idempotence and deduplication across devices, and how to design the schema to support later joins and metric computation.

MediumTechnical

0 practiced

Given an exposures table with potential duplicates and nulls:

experiment_exposures(exposure_id PK, user_id, experiment_id, variant, timestamp)

Write an ANSI SQL query that returns daily deduplicated exposure counts per variant (by date of first exposure per user), flags days where daily traffic deviates by >20% from a rolling 7-day median, and returns the top 3 anomalous dates per experiment_id.

Unlock Full Question Bank

Get access to hundreds of Experiment Design and Execution interview questions and detailed answers.

Join thousands of developers preparing for their dream job.