Statistical Inference and Hypothesis Testing Questions

Reasoning about uncertainty in data and drawing formal conclusions from samples. Covers probability rules and common distributions, the Central Limit Theorem, sampling, standard error, confidence intervals, and Bayesian reasoning, together with the significance-testing framework: null and alternative hypotheses, p-values, statistical power, Type I and Type II errors, effect sizes, and choosing the right test (t-test, chi-square, non-parametric). Emphasizes correctly interpreting statistical results and avoiding common misreadings of significance in business and product contexts rather than memorizing formulas.

MediumTechnical

33 practiced

Write a Python function that computes required sample size per group for a two-sided two-proportion z-test (normal approximation). Inputs: baseline_rate (p1), minimum_detectable_absolute_increase (delta), alpha, power. Show the formula and compute the required sample size per group for baseline_rate=0.10, delta=0.01 (absolute), alpha=0.05, power=0.80. Explain limitations of the normal approximation for small p.

Sample Answer

Approach / formulaFor a two-sided two‑proportion z‑test (normal approximation) with baseline p1 and target p2 = p1 + delta, the usual sample size per group is:

n = [ z_{1-α/2} * sqrt(2 * p̄ * (1 - p̄)) + z_{power} * sqrt(p1*(1-p1) + p2*(1-p2)) ]^2 / delta^2

where p̄ = (p1 + p2)/2, z_{q} is the q quantile of the standard normal (e.g., z_{1-α/2}=1.96 for α=0.05) and delta is the absolute difference p2-p1.

Python implementation (uses statistics.NormalDist for z-scores):

python

from statistics import NormalDist
from math import sqrt, ceil

def sample_size_two_prop(p1, delta, alpha=0.05, power=0.8):
    p2 = p1 + delta
    pbar = (p1 + p2) / 2
    z_alpha = NormalDist().inv_cdf(1 - alpha/2)
    z_power = NormalDist().inv_cdf(power)
    term1 = z_alpha * sqrt(2 * pbar * (1 - pbar))
    term2 = z_power * sqrt(p1*(1-p1) + p2*(1-p2))
    n = (term1 + term2)**2 / (delta**2)
    return int(ceil(n))

# Compute given values
p1 = 0.10
delta = 0.01
alpha = 0.05
power = 0.80

n_per_group = sample_size_two_prop(p1, delta, alpha, power)
n_per_group

Computed result (for p1=0.10, delta=0.01, α=0.05, power=0.80): about 14,756 subjects per group.

Limitations of the normal approximation- It assumes the sampling distribution of sample proportions is approximately normal; this can fail when p is very small or very large or when expected counts n*p or n*(1-p) are < ~5–10.- For small p or small n, the approximation underestimates tail probabilities; use exact/binomial-based methods, Fisher’s exact, or simulation-based power, or use more accurate formulas (e.g., using Wilson score) or statsmodels/statspower packages for better accuracy.

HardTechnical

29 practiced

When running many experiments and tests, when is it preferable to control false discovery rate (FDR) instead of family-wise error rate (FWER)? Give two business scenarios where FDR is more appropriate and two where FWER is required. Mention the statistical methods you would use for each.

EasyTechnical

25 practiced

Explain Type I and Type II errors using a concrete A/B test example: suppose you test a new checkout flow. Define both error types, describe the real-world business consequence of each, and explain how alpha and beta relate to those consequences.

MediumTechnical

24 practiced

Compare Cohen's d, absolute lift, relative risk (ratio), and odds ratio as effect size measures. For a binary business KPI such as conversion, explain which measure you would report to executives and how to compute and interpret it in plain language.

Sample Answer

Cohen’s d, absolute lift, relative risk (ratio), and odds ratio are all effect-size measures but answer different questions and suit different audiences for a binary KPI like conversion.

Short definitions and computation (binary conversion: control conversion p1, treatment p2):- Absolute lift (risk difference): p2 − p1. Simple percent-point change (e.g., 0.12 − 0.10 = +0.02 = +2 percentage points). Best for executives because it says “this campaign adds 2 extra conversions per 100 visitors.”- Relative risk (risk ratio): p2 / p1. Shows proportional change (e.g., 0.12 / 0.10 = 1.2 → 20% increase). Good secondary metric to communicate proportional impact.- Odds ratio (OR): (p2/(1−p2)) / (p1/(1−p1)). Compares odds, not probabilities; for p small they approximate RR, but diverge as p grows. Often used in logistic regression; can confuse non-technical stakeholders.- Cohen’s d: (mean2 − mean1) / pooled SD. For binary outcomes, means = proportions and SD depends on p(1−p). Cohen’s d translates difference into standard-deviation units — helpful for comparing across different KPIs but less intuitive for business stakeholders.

Which to report to executives and why:- Primary: absolute lift (risk difference). It’s concrete, actionable, and directly ties to business impact (extra conversions, revenue).- Secondary: relative risk for context (percent increase).- Avoid reporting odds ratio to executives unless they understand odds; if you must, translate OR into approximate RR or absolute lift.- Use Cohen’s d only when comparing effect sizes across heterogeneous metrics or when audiences care about standardized magnitudes.

Plain-language interpretation examples:- Absolute lift: “We increased conversion by 2 percentage points — from 10% to 12%. For every 1,000 visitors, that’s 20 more conversions.”- Relative risk: “That’s a 20% increase in conversion vs. control.”- Odds ratio (if needed): “The odds of converting are 1.22 times higher with the change; this roughly corresponds to a 20% higher conversion probability here.”- Cohen’s d: “The change equals 0.2 SD — a small effect compared to typical benchmarks.”

Best practices:- Always report baseline p1, sample sizes, and confidence intervals for the effect size.- Translate percentages to absolute counts and revenue impact for executives.- If using regression, report adjusted absolute lift when controlling for covariates.

MediumTechnical

50 practiced

Implement a Python function to compute statistical power for a two-sample t-test given: effect_size (Cohen's d), n_per_group, alpha (two-sided). You may use scipy.stats if desired; if not, outline the math. Then compute power for effect_size=0.3, n_per_group=200, alpha=0.05.

Sample Answer

Approach: For a two-sample (independent) t-test with equal sample sizes, compute noncentral t parameter delta = d * sqrt(n / 2) where d = Cohen's d and n = n_per_group. Power = 1 - beta = probability that |t| exceeds critical t under the noncentral t distribution. We can compute with scipy.stats.nct or use normal approximation for large n.

python

import math
from scipy import stats

def power_two_sample_t(effect_size, n_per_group, alpha=0.05):
    """
    Two-sided independent t-test power for equal n.
    effect_size: Cohen's d
    n_per_group: sample size per group
    alpha: two-sided significance level
    """
    df = 2 * n_per_group - 2
    # noncentrality parameter
    delta = effect_size * math.sqrt(n_per_group / 2.0)
    # critical t for two-sided test
    t_crit = stats.t.ppf(1 - alpha / 2, df)
    # power = P(T > t_crit) + P(T < -t_crit) under noncentral t with df and ncp=delta
    power = stats.nct.sf(t_crit, df, delta) + stats.nct.cdf(-t_crit, df, delta)
    return power

print(power_two_sample_t(0.3, 200, 0.05))

Result (computed): approximately 0.683 — about 68.3% power for d=0.3, n=200 per group, alpha=0.05.

Notes:- For large n, a normal approximation can be used: delta ~ d*sqrt(n/2), critical z = norm.ppf(1-alpha/2), power ≈ 1 - Φ(z - delta) + Φ(-z - delta) (or implement directly).- Edge cases: very small n (df small) where nct is needed; invalid inputs (n<2, d extremely large).

Unlock Full Question Bank

Get access to hundreds of Statistical Inference and Hypothesis Testing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.