Metrics, Guardrails, and Evaluation Criteria Questions
Design appropriate success metrics for experiments. Understand primary metrics, secondary metrics, and guardrail metrics. Know how to choose metrics that align with business goals while avoiding unintended consequences.
MediumTechnical
0 practiced
A recommender model increases predicted purchase probability but causes lower average order value because it recommends low-priced high-conversion items. Define guardrail metrics that would capture this trade-off. Propose an evaluation approach (offline and online) and a multi-objective decision rule to decide deployment.
HardTechnical
0 practiced
Implement a simplified Kaplan-Meier estimator in Python for churn analysis. Signature: def kaplan_meier(durations: List[float], events: List[int]) -> Tuple[List[float], List[float], float] returning (times, survival_probabilities, median_survival_time). 'events' is 1 if event (churn) observed and 0 if censored. Handle ties and right-censoring in your implementation and document assumptions.
HardSystem Design
0 practiced
Design an executive-level dashboard for experiments that summarizes for each experiment: primary metric lift with 95% CI, key secondary metrics, guardrail flags, sample sizes, effect heterogeneity by key segments, and a recommended action. Describe the data model, APIs, caching strategy, and UX considerations to promote trust and clarity.
MediumTechnical
0 practiced
How would you detect instrumentation or measurement bugs that affect metric computation? Provide a practical checklist of sanity checks, canary tests, duplication checks, uniqueness counts, time-travel events, and cross-system joins to surface discrepancies. Include a plan for debugging and preventing recurrence.
MediumTechnical
0 practiced
Explain Type I and Type II errors and how they interact with guardrail monitoring when you track multiple guardrail metrics simultaneously. Describe methods to control false positives across multiple metrics (Bonferroni, Benjamini-Hochberg FDR, hierarchical testing) and trade-offs of each approach for experimentation.
Unlock Full Question Bank
Get access to hundreds of Metrics, Guardrails, and Evaluation Criteria interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.