Metrics Selection and Diagnostic Interpretation Questions

Addresses how to choose appropriate metrics and how to interpret and diagnose metric changes. Includes selecting primary and secondary metrics for experiments and initiatives, balancing leading indicators against lagging indicators, avoiding metric gaming, and handling conflicting signals when different metrics move in different directions. Also covers anomaly detection and root cause diagnosis: given a metric change, enumerate potential causes, propose investigative steps, identify supporting diagnostic metrics or logs, design quick experiments or data queries to validate hypotheses, and recommend remedial actions. Communication of nuanced or inconclusive results to non technical stakeholders is also emphasized.

HardTechnical

35 practiced

A third-party vendor provides an attribution dataset that conflicts with your internal metrics. Outline a detailed audit plan: reconciliation checks you would run (unique users, conversions by date/channel), timestamp and timezone alignment, deduplication checks, sampling differences, and manual log-sampling tests. List which statistical tests and visualizations you'd use to communicate discrepancies to the vendor.

MediumTechnical

32 practiced

You're designing an A/B test for a signup funnel with a baseline signup conversion of 5%, average monthly eligible traffic of 10M users, and you want 80% power with alpha=0.05 to detect a minimum detectable effect (MDE) of 0.5 percentage points (i.e., from 5.0% to 5.5%). Describe how you'd compute required sample size per variant, any assumptions you need to make, and practical constraints for running this experiment. Also name 3 guardrail metrics you'd monitor during the test.

EasyTechnical

38 practiced

Explain the difference between leading and lagging indicators in the context of a subscription SaaS product. Give three concrete examples of each (leading and lagging), explain why they fall into that category, and describe a situation where you'd prefer to act on a leading indicator instead of a lagging one.

EasySystem Design

31 practiced

You're tasked with building an executive dashboard that tracks Weekly Active Users (WAU) and retention for a global mobile product. Describe which metrics and sub-metrics you would include, precise definitions (time windows, activity criteria), cohort segmentation (by signup week, acquisition channel, geography), and suggested visualizations. Also explain how to compute WAU from an events table (schema provided below):

events(user_id bigint, event_time timestamp, event_name varchar, platform varchar)

Be explicit about timezone handling and windows.

MediumTechnical

43 practiced

Create a BI acceptance checklist for a newly defined metric that will be released to production dashboards. The checklist should include requirements for definition, owner, source-of-truth, transformation logic, automated tests, SLA/freshness, monitoring, and communication. Also propose two ways to automate health checks for this metric nightly.

Unlock Full Question Bank

Get access to hundreds of Metrics Selection and Diagnostic Interpretation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.