Statistical Inference and Hypothesis Testing Questions

Direct answer

A 95% confidence interval is a range built from your sample using a procedure that, if repeated many times, would produce an interval containing the true, fixed, unknown parameter about 95% of the time. It is a statement about the reliability of the procedure across hypothetical repetitions, not a 95% probability that this one specific, already-computed interval contains the true value. The true value either is or isn't in this particular interval; there's no probability left to talk about once the interval is fixed.

Structured elaboration

The correct frequentist interpretation

Before data is collected, the interval's endpoints are random, since they depend on which sample happens to be drawn; the true parameter is a fixed, unknown constant. "95% confidence" describes the long-run behavior of the interval-producing procedure: across many hypothetical repeated samples, about 95% of the intervals it produces would bracket the true value, and about 5% would miss it. Once actual data is collected and one specific interval is computed, say [41.2, 48.7], that interval either contains the true value or it doesn't. The randomness was in the sampling and calculation, not in the now-fixed true parameter or the now-fixed interval.

The common misreading, and why it's wrong

"There's a 95% chance the true value is in this interval" sounds almost identical to the correct statement but treats the true parameter as the random thing with a 95% probability of being in some location, when in the frequentist framework the true parameter isn't random at all, only the sampling procedure is. This misreading is extremely common, surveys of both students and working researchers find majorities endorse it, precisely because the language is subtle: "confidence" was chosen specifically to avoid claiming "probability" about a fixed parameter, but in everyday usage the two words feel interchangeable.

Why the distinction matters practically

It matters most when someone tries to use a single computed interval to make a probabilistic claim. "There's a 95% chance the true conversion lift is between 2% and 6%" is not licensed by a frequentist confidence interval, that statement would require a Bayesian credible interval, which is a genuine probability statement about the parameter given a prior. Swapping one framework's language onto the other's number is a real, if common, error, not just pedantry; it can lead to overconfident decisions when the sampling assumptions behind the interval, random sampling, correct variance model, are shakier than they look.

Plain-language version for a non-statistician

"We can't know the exact true number, only estimate it from our sample. This range is our best estimate plus a margin that accounts for sampling luck: if we ran this same measurement over and over on different random samples, about 19 times out of 20 the range we'd compute would contain the real number. So this range is the set of values consistent with what we observed, not a guarantee, but a calibrated one."

Worked example

Suppose (illustrative numbers, chosen to demonstrate the interpretation, not derived from a specific dataset) an A/B test reports a 95% confidence interval for the lift in conversion rate of [1.2%, 5.8%], entirely above zero.

• Correct statement: using this measurement procedure, if the experiment were rerun many times, about 95% of the resulting intervals would contain the true lift; this particular interval happens to be [1.2%, 5.8%].
• Correct practical use: because the interval excludes 0, the data are inconsistent with "no effect" at the 5% significance level, equivalent to a two-sided test rejecting the null of zero lift at alpha = 0.05.
• Incorrect statement to avoid: "there's a 95% probability the true lift is between 1.2% and 5.8%," which treats the fixed true lift as the random quantity.
• Incorrect statement to avoid: "95% of future users will see a lift in this range," which confuses a confidence interval for the mean parameter with a prediction interval for individual outcomes, a different and typically much wider interval.

Trade-offs and pitfalls

• The CI-versus-hypothesis-test duality, excludes zero if and only if significant at that alpha, is genuinely useful, but leaning on it too heavily can obscure that the interval also communicates magnitude and precision, which a bare p-value doesn't. Report the interval, not just significant or not significant.
• A narrow interval is not automatically good news; it just means the estimate is precise. A narrow interval tightly around a practically meaningless effect size is still a practically meaningless result.
• All of this assumes the sampling and modeling assumptions behind the interval are valid, random or representative sampling, correctly specified variance. A confidence interval computed on a biased or non-random sample gives a precise-looking interval around the wrong quantity; precision is not the same as accuracy.
• When talking to non-technical stakeholders, resist compressing the explanation down to "95% chance it's in this range," that's exactly the misreading. Use the "if we repeated this, most of the ranges we'd get would contain the truth" framing instead, even though it takes a few more words.

Conditional probability updates a belief about an event given new evidence; Bayes' theorem does this formally by combining the detector's known accuracy with how rare fraud actually is. Even with a 98% true positive rate and only a 1% false positive rate, a 0.1% base rate of fraud means most flagged transactions are still false alarms: the posterior probability a flagged transaction is truly fraudulent comes out to about 8.9%.

Structured elaboration

The formula

$P(\text{Fraud}\mid \text{Flag}) = \frac{P(\text{Flag}\mid \text{Fraud})\,P(\text{Fraud})}{P(\text{Flag}\mid \text{Fraud})\,P(\text{Fraud}) + P(\text{Flag}\mid \text{Not Fraud})\,P(\text{Not Fraud})}$

Each term has a name:

• P(Fraud): the prior, the base rate before seeing the flag.
• P(Flag given Fraud): the likelihood, here the true positive rate (sensitivity).
• P(Flag given Not Fraud): the false positive rate.
• The denominator, P(Flag), is the total probability of a flag from either a truly fraudulent or a truly legitimate transaction. This is what "explains away" most of the apparent accuracy.

Why intuition misleads here (base-rate neglect)

It is tempting to read "98% true positive rate" as "a flag is 98% likely to be fraud." That conflates P(Flag given Fraud) with P(Fraud given Flag), which are different conditional probabilities. When the event being detected is rare, even a small false positive rate produces far more false alarms than true ones in absolute terms, because there are so many more legitimate transactions to draw false positives from.

Worked example

$P(\text{Fraud}\mid \text{Flag}) = \frac{0.98 \times 0.001}{0.98 \times 0.001 + 0.01 \times 0.999} = \frac{0.00098}{0.01097} \approx 0.0893$

So about 8.9% of flagged transactions are actually fraudulent; roughly 91% are false alarms (computed directly from the formula above, no simulation needed).

Implications for the decision threshold: the false positive rate drives the posterior almost as much as the base rate does, because it applies to the much larger pool of legitimate transactions. Holding the true positive rate fixed at 98% and varying only the false positive rate:

(each row computed from the same formula with only the false positive rate changed; verified with python3)

Trade-offs & pitfalls

• A model can have excellent sensitivity and still produce a low-precision alert stream when the target event is rare; sensitivity alone is not enough to judge a detector meant for a rare class.
• Tightening the threshold to shrink the false positive rate raises the posterior (fewer false alarms per true one), but it comes at the cost of missing more true fraud (lower true positive rate); the two move in opposite directions.
• Operationally, an 8.9% posterior does not mean "ignore the flag." It means route flagged transactions to a cheaper secondary check (rules, a human reviewer, step-up authentication) rather than an automatic block, so the cost of a false positive stays low.
• Base rates drift over time as fraud patterns and volumes change; a posterior computed once goes stale if the prior is not periodically re-estimated.

Direct answer

Revenue up while retention and NPS decline for two quarters is a classic sign that the business is winning short-term monetization at the expense of long-term health, most often through mix shift (different, sometimes lower-fit users or a different pricing structure) rather than the product genuinely improving. The investigation should treat "revenue growth" and "retention/NPS decline" as two outcomes potentially driven by a shared, upstream cause (pricing, acquisition mix, or a product change), not evaluate them independently.

Structured elaboration

Hypotheses to test, roughly in order of how commonly they explain this exact pattern

1. Pricing or monetization change increased ARPU but also increased friction or perceived value mismatch, driving both the revenue bump and dissatisfaction.
2. Acquisition mix shifted toward higher-paying but lower-fit users (e.g. a paid channel or enterprise push brought in users who convert to revenue quickly but churn or complain more).
3. A specific feature or experiment increased short-term monetization while degrading the core experience for a subset of users.
4. One-time or concentrated revenue events (large enterprise deals, a promotion) inflate aggregate revenue while the core, ongoing user base's underlying health is actually flat or declining.
5. Measurement or attribution issue: revenue recognition change or duplicate counting inflating the topline number without a real behavioral change at all.

Metrics and cohorts to analyze

Statistical tests and analyses to run

• Cohort retention heatmap crossed with ARPU decile: is the revenue growth concentrated in cohorts whose retention is also declining, or in a separate, healthy cohort?
• Difference-in-differences comparing cohorts before and after the suspected driver (e.g. a pricing change), against a comparable unaffected baseline, to isolate its effect on both revenue and retention.
• Regression of retention on covariates including acquisition channel and revenue tier, to test whether channel or tier explains the retention decline once controlled for, versus it being broad-based across the whole user base.
• Trend significance test on the NPS and retention time series themselves (e.g. comparing quarter-over-quarter means with appropriate variance estimates) to confirm the decline isn't within normal seasonal noise before treating it as a real signal worth acting on.

Decisions that might follow

Worked example

Suppose a cohort retention heatmap crossed with ARPU decile shows that the top 20% of ARPU users (by revenue) have 30-day retention of 38%, versus 61% for the bottom 80%. Splitting further by acquisition channel shows the top-ARPU decile is 70% sourced from a single paid channel launched two quarters ago, versus that channel being only 15% of the broader user base. This pattern (a new, concentrated acquisition channel simultaneously driving the ARPU decile up and dragging blended retention down) points toward hypothesis 2: the revenue growth and the retention/NPS decline share a common cause in acquisition mix, rather than the product itself degrading for its existing base. That reframes the fix as a channel-quality and targeting problem, not a product-quality problem, though the team should still verify NPS specifically within that channel's cohort to rule out a genuine experience gap for those users too.

Trade-offs & pitfalls

• Don't average away the story. A blended, company-wide retention number can look like a modest decline while masking a severe drop in one segment offset by stability elsewhere; always cross revenue and retention cuts by the same cohort dimensions.
• Correlation between the revenue and retention trends is not proof of a shared cause. Test the specific hypothesized mechanism (e.g. via difference-in-differences around the suspected driver) rather than asserting a link because two lines moved in opposite directions in the same period.
• NPS is a noisy, low-response-rate metric. A shift in who responds to the survey (not just how they feel) can produce an apparent decline; check response rate and respondent composition alongside the score itself.
• A quarter or two of concentrated enterprise deals is easy to over-interpret as "the business is healthy." Report revenue concentration (e.g. share from the top 10 accounts) alongside the headline number so a one-time deal isn't mistaken for durable growth.
• Acting on hypothesis 1 (pricing) versus hypothesis 2 (acquisition mix) implies very different fixes, so resist moving to remediation before the cohort-level analysis actually distinguishes between them; treating a mix-shift problem as a product or pricing problem wastes a cycle and doesn't fix the underlying issue.

Direct answer

Three standard variance-reduction techniques for a noisy revenue-per-user metric are CUPED (covariate adjustment using a pre-experiment value of the metric), stratification/blocking on a variable predictive of the outcome, and outcome transformation (e.g. log or winsorizing to tame heavy tails). All three reduce the standard error of the treatment effect estimate without touching the randomization, which means they let you detect the same effect with fewer users, or detect a smaller effect with the same sample size.

Structured elaboration

1) CUPED (covariate adjustment using a pre-period covariate).

• Assumption: the covariate (e.g. the same user's revenue in the weeks before the experiment) is measured pre-treatment, so adjusting on it cannot introduce bias, and it correlates with the outcome.
• Implementation: compute $\theta = \text{Cov}(Y, C) / \text{Var}(C)$ on the pooled sample, then form the adjusted outcome $Y' = Y - \theta(C - \bar{C})$, and run the usual treatment-effect test on $Y'$ instead of $Y$.
• Expected effect on variance: the variance reduction is exactly $\rho^2$, the squared correlation between the covariate and the outcome; a correlation of 0.4 gives a 16% variance reduction, a correlation of 0.7 gives about 49%.
• Pitfalls: using a covariate measured after treatment starts (even a mediator that happens to look pre-treatment) can leak treatment effect into the adjustment and bias the estimate; a weak pre-period correlation gives little benefit for the extra pipeline complexity.

2) Stratification / blocking.

• Assumption: the stratifying variable (cohort, geography, acquisition channel) is strongly predictive of the outcome and, critically, is not itself affected by treatment.
• Implementation: pre-specify strata, randomize within each stratum (not across the whole population), then compute a stratum-weighted treatment effect.
• Expected effect on variance: reduces the between-strata component of variance; the more of the outcome variance the stratifying variable explains, the bigger the gain.
• Pitfalls: too many small strata reduces power per stratum and complicates aggregation; stratifying post hoc after seeing results is a form of p-hacking, not a variance-reduction technique.

3) Outcome transformation (log / winsorizing).

• Assumption: the transformation stabilizes variance without destroying the causal contrast you care about, or you're willing to interpret the result on the transformed scale.
• Implementation: apply $\log(Y + c)$ for heavy right tails, or winsorize at a chosen percentile (cap the top/bottom 1-5% of values) before differencing means.
• Expected effect on variance: reduces the outsized influence of a handful of extreme spenders on the sample variance, which is often the single biggest driver of noise in revenue-per-user metrics.
• Pitfalls: the choice of the additive constant $c$ in a log transform, or the winsorizing percentile, is itself a researcher degree of freedom that should be pre-specified; truncating too aggressively can bias the mean estimate if high spenders are exactly who the treatment is meant to affect.

Worked example

Simulating revenue data with a realistic right-skewed shape and a pre-period covariate correlated with the outcome (pinned seed, numpy.random.default_rng(1134), n = 20,000 users):

import numpy as np
rng = np.random.default_rng(seed=1134)
n = 20000
propensity = rng.gamma(shape=2.0, scale=1.0, size=n)
pre_revenue = propensity * rng.gamma(shape=2.0, scale=4.0, size=n)
post_revenue = propensity * rng.gamma(shape=2.0, scale=4.2, size=n) + rng.normal(0, 1, n).clip(min=-5)

theta = np.cov(post_revenue, pre_revenue, ddof=1)[0, 1] / np.var(pre_revenue, ddof=1)
y_adj = post_revenue - theta * (pre_revenue - pre_revenue.mean())

var_raw = np.var(post_revenue, ddof=1)
var_adj = np.var(y_adj, ddof=1)
reduction = 1 - var_adj / var_raw

This produces a correlation between pre- and post-period revenue of $\rho = 0.380$, $\theta = 0.405$, raw variance of 341.25, CUPED-adjusted variance of 291.99, and a variance reduction of 14.4%, matching the theoretical $\rho^2 = 0.380^2 = 14.4\%$ prediction exactly. Since the required sample size for a fixed power scales with the variance, that 14.4% variance reduction translates to a standard-error shrink factor of $\sqrt{1 - 0.144} = 0.925$, meaning the same statistical power is reached at roughly 86% of the original sample size (equivalently, the experiment could run about 14% fewer users, or the same users for a shorter time), for identical power. The adjustment left the mean unchanged (16.8395 before and after), confirming CUPED does not bias the point estimate, only its precision.

Trade-offs & pitfalls

Variance reduction techniques are unbiasedness-preserving by design when applied correctly (adjusting only on pre-treatment covariates, pre-specifying strata), but each introduces a new failure mode if applied carelessly: a leaked post-treatment covariate in CUPED, post-hoc strata chosen to flatter the result, or an aggressively chosen winsorizing threshold that quietly changes what population the effect estimate describes. In practice CUPED is usually the highest-leverage of the three when a stable pre-period metric exists, because it requires no change to the experiment design itself (just an adjustment at analysis time), whereas stratification requires committing to the stratification variable before randomization begins.

Direct answer

An analysis that only shows the significant subgroup results, with no mention of how many subgroups were tried, is a textbook signature of p-hacking or selective reporting. Audit it by reconstructing the full set of comparisons that were actually run (not just the ones reported), re-testing that full set with a multiplicity correction, and re-running the analysis end to end on the raw data. The reanalysis plan should pre-specify a small, justified set of primary comparisons and report the complete picture, significant and non-significant alike, not a curated slice of it.

Structured elaboration

Concrete audit checks

Robust reanalysis plan

1. Reproduce the original result from raw data and code before doing anything else.
2. List every comparison that was actually run, whether or not it appeared in the report.
3. Apply a multiplicity correction (Bonferroni for a strict guarantee, Benjamini-Hochberg for a more powered but still principled one) across that full list, not just the reported subset.
4. Report the complete table: every tested subgroup, its unadjusted and adjusted p-value, its effect size, and its sample size, so nothing is hidden by omission.
5. Clearly separate confirmatory findings (survive correction) from exploratory ones (interesting, but not proven, and worth a dedicated follow-up test).

Worked example

Suppose 12 subgroup comparisons were actually run internally, but the report only surfaces the one with $p=0.031$. Re-testing the full family of 12 with Holm-Bonferroni (step-down, more powerful than flat Bonferroni but still FWER-controlling, meaning it keeps the family-wise error rate, the chance that even one of the reported findings is a false positive, at or below the target rate across all 12 comparisons, the same guarantee flat Bonferroni gives, just with more power):

(with $m=12$ and $\alpha=0.05$; computed directly from the Holm step-down formula)

The smallest p-value in the full family, 0.031, needed to clear 0.00417 to survive Holm-Bonferroni; it doesn't. The "significant" subgroup finding that made it into the report does not survive once the other 11 comparisons that were actually run are accounted for. This is exactly the pattern an audit is designed to catch, and it doesn't require re-collecting any data, only re-testing honestly against the full family.

Trade-offs & pitfalls

• The hardest part of this audit is usually not the statistics, it's getting an honest accounting of how many comparisons were actually tried; without that, no correction can be computed correctly, so insist on the code and logs, not just a verbal assurance.
• Correcting for multiplicity can occasionally kill a genuinely real effect along with the false ones; that's the acknowledged cost of FWER or FDR control, not a reason to skip it, but it's worth flagging any borderline case as "worth a dedicated confirmatory follow-up" rather than dismissing it outright.
• Presenting this audit to the original analysts or to leadership needs care: the goal is fixing the process (pre-registration, full reporting), not accusing anyone of misconduct, since selective reporting this way is frequently an unconscious byproduct of exploratory analysis rather than deliberate manipulation.
• A reanalysis that only tightens the statistics but keeps reporting only the "winning" comparisons repeats the original mistake in a more sophisticated wrapper; the fix has to include reporting the full family, not just applying a correction to the subset that was already selected.