Experimentation Metrics and Strategy Questions

Designing experiments and selecting appropriate primary, secondary, and guardrail metrics to evaluate hypotheses while protecting long term user value. This includes choosing metrics that reflect both short term signal and long term outcomes, reasoning about metric interactions and potential unintended consequences, and applying statistical considerations such as minimum detectable effect, sample size and power analysis, test duration, and external validity across segments and platforms. Candidates should also discuss experiment risk mitigation, stopping rules, and how to operationalize experiment results into product decisions.

MediumTechnical

0 practiced

Write a Python script to detect Sample Ratio Mismatch (SRM) for a simple two-variant experiment using a chi-squared test. Input: counts dictionary {'control': n1, 'treatment': n2}. Output: observed proportion, expected proportion, chi2 statistic, and p-value. Include input validation and an example run.

MediumTechnical

0 practiced

Explain sequential testing and why repeatedly peeking at p-values inflates type I error. Provide two practical stopping rules an engineering team could implement and discuss the operational trade-offs (complexity, speed, and safety).

EasyTechnical

0 practiced

You are responsible for event instrumentation for experiments. Provide a short checklist (5-8 items) that ensures event quality for experimentation: include idempotency, deduplication, stable user IDs, timestamps, and schema versioning. Explain why each item matters and how you'd test them post-deploy.

HardTechnical

0 practiced

Explain the Sequential Probability Ratio Test (SPRT) and outline how you would implement a sequential test for a continuous metric in an experimentation platform. Discuss alpha control, expected sample path behavior, and how you would simulate operating characteristics before productionizing.

HardTechnical

0 practiced

Describe an attribution strategy for experiments that change multiple funnel stages (impression → click → purchase). How would you attribute the net effect on final purchases to intermediate metric changes? Mention causal mediation analysis or incremental contribution approaches and how you'd compute this in practice.

Unlock Full Question Bank

Get access to hundreds of Experimentation Metrics and Strategy interview questions and detailed answers.

Join thousands of developers preparing for their dream job.