Feature Success and A/B Testing Questions

How you'd measure success of a specific feature launch. Setting up experiments or A/B tests. Understanding statistical significance and sample sizes at a basic level. Interpreting results and deciding when to ship, iterate, or kill a feature.

EasyTechnical

77 practiced

Describe Minimum Detectable Effect (MDE) and why it's important when planning experiments. Provide an example calculation narrative: baseline conversion = 4%, desired MDE = 0.6% absolute (i.e., 15% relative), alpha 0.05, power 0.8 — explain how MDE affects sample size and experiment duration in words (no formulas required).

EasyTechnical

67 practiced

List and explain 4 visualization types that are helpful when presenting experiment results (e.g., time series, bar chart with error bars). For each visualization explain what it emphasizes and one pitfall to avoid when interpreting experiment data.

MediumTechnical

65 practiced

Discuss common threats to causal inference in A/B tests (e.g., contamination, non-compliance, interference, differential attrition). For each threat, give a concrete mitigation you would implement as a BI analyst before or during analysis.

MediumTechnical

84 practiced

You run many experiments and evaluate dozens of metrics. Explain multiple-comparisons and false-discovery problems in that context. Describe two statistical approaches (one conservative, one that controls false discovery rate) you could implement in your analysis pipeline to limit false positives.

HardTechnical

70 practiced

Describe how to set up an experiment metric SLO (Service Level Objective) for critical business metrics (e.g., revenue per user) so experiments do not degrade the business beyond an agreed threshold. Include how you'd implement automated blocking logic, monitoring cadence, and process for overruling an SLO.

Unlock Full Question Bank

Get access to hundreds of Feature Success and A/B Testing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.