Metrics, Guardrails, and Evaluation Criteria Questions
Design appropriate success metrics for experiments. Understand primary metrics, secondary metrics, and guardrail metrics. Know how to choose metrics that align with business goals while avoiding unintended consequences.
EasyTechnical
0 practiced
What is a guardrail metric? Provide three concrete examples across product usability, monetization, and system reliability. Explain why each is appropriately a guardrail (not primary/secondary) and how you would instrument and monitor it.
MediumTechnical
0 practiced
Given event tables:Write SQL to compute 7-day retention per variant where retention is defined as at least one event in day 7 since first exposure. Return variant, day_0_users, day_7_retained, retention_rate, and a rough 95% CI for the retention rate (e.g., Wilson). Assume event_time is timestamp and first exposure date is the min event_time per user.
sql
events(user_id, variant, event_type, event_time)
users(user_id, signup_date)MediumTechnical
0 practiced
You observe a 10% increase in clicks but a 5% decrease in purchases after a UI change. Outline an investigative plan: which metrics to compute, how to segment users, what causal checks to run, potential confounders, and experiments or analyses you'd run next to identify root cause and decide on action.
MediumTechnical
0 practiced
A recommender model increases predicted purchase probability but causes lower average order value because it recommends low-priced high-conversion items. Define guardrail metrics that would capture this trade-off. Propose an evaluation approach (offline and online) and a multi-objective decision rule to decide deployment.
EasyTechnical
0 practiced
Define primary, secondary, and guardrail metrics in the context of product experimentation. Provide one concrete example of each using an e-commerce A/B test that changes product image size and layout. Explain why each metric belongs to its category and how it maps to a business objective and potential harms to watch for.
Unlock Full Question Bank
Get access to hundreds of Metrics, Guardrails, and Evaluation Criteria interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.