A/B Testing and Optimization Methodology Questions
Discuss your experience designing and running A/B tests on content elements: headlines, formats, messaging, calls-to-action, visual design, content length, etc. Share specific examples of tests you've run with results and how you implemented learnings. Discuss statistical significance and proper experimental design. Show how you prioritize testing opportunities and build a testing roadmap.
MediumTechnical
0 practiced
Explain how unequal traffic allocation (for example 10% control and 90% treatment) affects statistical power and sample size requirements. Provide the intuition and a brief formulaic approach to compute adjusted sample sizes for a binary outcome under allocation ratio r.
HardTechnical
0 practiced
Explain how to perform offline counterfactual policy evaluation for a new ranking or recommendation policy using logged bandit feedback. Discuss IPS, self-normalized IPS (SNIPS), and doubly-robust (DR) estimators, assumptions required (propensities, support), weight clipping, and variance reduction techniques. Describe how you'd validate offline estimates before an online rollout.
EasyTechnical
0 practiced
Explain consistent bucketing (deterministic assignment) for A/B tests across microservices. Why is hashing a user ID important, what are common pitfalls (e.g., cookie deletion, different namespaces), and how do you handle cases like anonymous users, bots, or session-based users?
MediumTechnical
0 practiced
Compare multi-armed bandits and classic A/B testing for content optimization. Discuss trade-offs in speed to choose a better variant, cumulative regret, interpretability of results, and business risk. Give concrete scenarios where bandits are preferable and where traditional A/B testing remains the better choice.
EasyTechnical
0 practiced
What is Sample Ratio Mismatch (SRM) in A/B testing? Explain how you would detect SRM in an experiment, list likely causes (instrumentation bugs, biased assignment, cookie loss), and outline immediate steps and mitigations you would take when SRM is observed.
Unlock Full Question Bank
Get access to hundreds of A/B Testing and Optimization Methodology interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.