Feature Success and A/B Testing Questions
How you'd measure success of a specific feature launch. Setting up experiments or A/B tests. Understanding statistical significance and sample sizes at a basic level. Interpreting results and deciding when to ship, iterate, or kill a feature.
HardTechnical
0 practiced
Compare Bayesian A/B testing to frequentist A/B testing from the standpoint of a BI analyst. Discuss interpretation differences, handling of peeking, prior selection, and how each approach affects decision thresholds for shipping features.
HardTechnical
0 practiced
You observe a feature produces a small but statistically significant lift in short-term engagement metrics while causing a large drop in long-term retention after 30 days. As a BI analyst, outline a step-by-step analytical plan to investigate root causes and decide whether to ship, iterate, or kill the feature. Include cohort and survival analysis ideas.
MediumTechnical
0 practiced
You observe a treatment has heterogeneous effects across user segments (e.g., new vs returning users). Describe statistical approaches to quantify heterogeneous treatment effects and how you'd present results to product and growth teams. Mention at least one machine-learning approach and one regression-based approach.
EasyTechnical
0 practiced
Explain Type I and Type II errors in the context of product experiments. Give an example cost for each error type for a subscription-based product and explain how alpha and power settings relate to the business trade-offs.
HardSystem Design
0 practiced
Architect an enterprise experiment platform to support thousands of concurrent experiments, enforce metric-definition versioning, detect cross-experiment conflicts, and provide experiment lineage for audits. Describe key components, data model for experiment metadata, how to prevent accidental overlapping treatments, and trade-offs between automation and safety.
Unlock Full Question Bank
Get access to hundreds of Feature Success and A/B Testing interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.