InterviewStack.io LogoInterviewStack.io

Experiment Design Analysis and Causal Methods Questions

Design and analysis of experiments and causal inference methods for when randomization is not possible. Candidates should know strategies to ensure randomization and evaluate experiment quality compute sample size and minimum detectable effect select and interpret primary and guardrail metrics and design appropriate test duration. Analysis skills include hypothesis testing p values confidence intervals effect size estimation variance estimation and variance reduction segmentation and interaction analysis and robust reporting of uncertainty. This topic covers observational and quasi experimental approaches such as propensity score matching difference in differences and regression discontinuity how to reason about confounding and selection bias and when to prefer a quasi experimental approach over a randomized test. Candidates should be able to translate causal conclusions into actionable guidance recommend follow up analyses and triangulate evidence across methods.

HardTechnical
27 practiced
Design and implement a sensitivity analysis to assess how robust an estimated Average Treatment Effect is to unmeasured confounding. Describe methods like Rosenbaum bounds and E-values, how to compute them at scale, and how to include these diagnostics in automated experiment reports to inform business decision-making.
EasyTechnical
30 practiced
Explain how blocking or stratified randomization reduces variance of treatment effect estimates. As a data engineer, describe how you would implement stratified assignment (e.g., by country or device type) and compute stratified effect estimates and combined variance in the metric pipeline.
MediumTechnical
24 practiced
Describe how to detect and handle Sample Ratio Mismatch (SRM) during an experiment. Provide a prioritized troubleshooting checklist including assignment salt changes, instrumentation mismatch, cross-environment traffic, and rollouts. Suggest concrete SQL queries or logs you would inspect to find the root cause.
MediumTechnical
26 practiced
Explain propensity score matching (PSM) end-to-end: how to build and validate the propensity model, choose matching algorithm (nearest neighbor, caliper), run diagnostics (standardized mean differences, overlap), and implement PSM at scale using Spark including how to persist propensity scores and matched pairs for reproducibility.
MediumTechnical
31 practiced
Describe a difference-in-differences (DiD) design. Given a panel table (user_id, date, revenue, treated_group_flag, post_period_flag), write the regression specification you would use (including group and time fixed effects) to estimate the DiD effect, and list key identifying assumptions and diagnostics you would run.

Unlock Full Question Bank

Get access to hundreds of Experiment Design Analysis and Causal Methods interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.