Metrics Selection and Diagnostic Interpretation Questions

Addresses how to choose appropriate metrics and how to interpret and diagnose metric changes. Includes selecting primary and secondary metrics for experiments and initiatives, balancing leading indicators against lagging indicators, avoiding metric gaming, and handling conflicting signals when different metrics move in different directions. Also covers anomaly detection and root cause diagnosis: given a metric change, enumerate potential causes, propose investigative steps, identify supporting diagnostic metrics or logs, design quick experiments or data queries to validate hypotheses, and recommend remedial actions. Communication of nuanced or inconclusive results to non technical stakeholders is also emphasized.

MediumTechnical

0 practiced

A product change may have reduced ARPU. Design a set of diagnostic metrics and logs you would create or query to triage the drop in ARPU. Include SQL-level queries to compute ARPU by cohort, by plan, and by acquisition channel, and list logs/sources you would check (billing service, checkout events, promo codes). Also propose 2 quick experiments to validate hypotheses.

HardTechnical

0 practiced

A metric drop correlates closely in time with a backend deployment. Describe how you would validate causation vs correlation. Propose quick experiments or analyses (canary vs full rollout comparison, synthetic control, rollback for a small region) and define objective rollback criteria that balance business risk and user impact. Also list the logs and telemetry you would inspect first.

MediumTechnical

0 practiced

Write SQL or describe LookML measures to compute Monthly Recurring Revenue (MRR) for a billing system with proration, upgrades, downgrades, and subscription churn. Use sample schema:

subscriptions(subscription_id, user_id, plan_id, start_date, end_date, monthly_price, proration_factor)

Define clearly how you would prorate partial months, handle upgrades mid-month, and exclude canceled subscriptions not active in the month. Return MRR for a specified month.

MediumTechnical

0 practiced

Given a simplified events table: events(user_id, event_time, event_name), write an ANSI SQL query to compute daily funnel conversion for the following funnel steps: 'view_product' -> 'add_to_cart' -> 'begin_checkout' -> 'purchase'. For each day in the last 14 days return: date, unique_users_view, unique_users_add, unique_users_begin_checkout, unique_users_purchase, conversion_rate_view_to_purchase, and step_drop_off_percentages. Assume we care about unique users performing each step on that calendar day (UTC).

HardTechnical

0 practiced

You're responsible for preventing metric gaming across the organization. Describe a combined technical, process, and people strategy: include instrumentation checks, monitoring heuristics to detect gaming (distributional shifts, user-level anomalies), governance (change control, metric owners), and incentive alignment. Provide concrete examples of detection rules and controls.

Unlock Full Question Bank

Get access to hundreds of Metrics Selection and Diagnostic Interpretation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.