Metrics, Guardrails, and Evaluation Criteria Questions
Design appropriate success metrics for experiments. Understand primary metrics, secondary metrics, and guardrail metrics. Know how to choose metrics that align with business goals while avoiding unintended consequences.
MediumTechnical
0 practiced
Given tables:Write SQL (Postgres) to compute incremental revenue per user between treatment and control, grouped by country, with a 95% CI using bootstrapping or the delta method. Explain your approach in comments.
sql
users(user_id, country)
orders(order_id, user_id, amount, occurred_at)HardTechnical
0 practiced
A product team requests a composite metric that combines revenue, retention, and NPS into a single objective to optimize. Propose a rigorous approach to define this composite: normalization, weighting, guardrails to prevent gaming, and a validation plan to ensure alignment with long-term business value.
MediumTechnical
0 practiced
Compare absolute difference, relative difference, and standardized effect size (Cohen's d) for reporting experiment results. For each, explain interpretability, dependence on baseline, and scenarios where one is preferred over the others.
EasyTechnical
0 practiced
Define a guardrail metric and explain why guardrails are necessary even if the primary metric is improving. Give two concise examples of guardrails and what types of adverse effects they protect against.
HardTechnical
0 practiced
Discuss the trade-offs between regression-adjusted estimates (covariate adjustment) and pure randomized difference-in-means when analyzing treatment effects on continuous outcomes. Cover efficiency gains, assumptions required, potential biases, and when to prefer one approach over the other.
Unlock Full Question Bank
Get access to hundreds of Metrics, Guardrails, and Evaluation Criteria interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.