Statistical Inference and Hypothesis Testing Questions

Direct answer

Ratio metrics like revenue per user are dangerous because they collapse two moving parts (a numerator and a denominator) into one number, and because revenue-like numerators are almost always heavy-tailed: a handful of large purchases can dominate the mean and make an experiment look like a clear win or loss when the typical user didn't change at all. The fix is to report a robust summary alongside the mean (median, trimmed mean, or a capped/winsorized version), define the denominator precisely and consistently across arms, and segment before concluding "the metric moved."

Structured elaboration

Pitfalls of ratio metrics

Robust alternatives and preprocessing

• Median or other percentiles (P50, P90): insensitive to a handful of extreme values, at the cost of ignoring exactly how much the tail moved.
• Trimmed or winsorized mean: cap or remove the top X% before averaging; keeps more information than the median while limiting whale influence.
• Log-transform (e.g. $\log(1 + \text{revenue})$): compresses the tail and stabilizes variance for a mean-based test; interpret results multiplicatively (geometric-mean-like) rather than additively.
• Fixed, pre-registered denominator: define "user" identically in both arms (e.g. "assigned at randomization," not "active during the window") so the denominator can't drift with treatment-induced behavior.

Analyzing and communicating results

1. Pre-specify the primary metric and its exact numerator/denominator/transform before the experiment runs.
2. Report mean, median, and a capped/winsorized mean together, plus a bootstrap CI for each (since ratio-metric sampling distributions are rarely well approximated by a simple closed-form CI).
3. Check whether the treatment changed the user mix (e.g. share of new vs. returning users); if so, present both the raw aggregate and a mix-adjusted (segment-weighted) comparison.
4. In the writeup, lead with whichever statistic best reflects the decision at hand, but always show all of them side by side so a stakeholder can see whether a "win" is broad-based or driven by a few large accounts.

Worked example

Simulated per-user revenue in a control arm (mostly small or zero purchases), and a treatment arm identical except for one added large ("whale") purchase, pinned seed:

import numpy as np
rng = np.random.default_rng(seed=99)

n = 1000
control = rng.choice([0, 0, 0, 5, 10, 20], size=n,
                      p=[0.5, 0.2, 0.1, 0.1, 0.06, 0.04]).astype(float)
treatment = control.copy()
rng.shuffle(treatment)
treatment[0] += 5000  # one large purchase added

mean_c, median_c = control.mean(), np.median(control)   # 2.195, 0.0
mean_t, median_t = treatment.mean(), np.median(treatment)  # 7.195, 0.0

lift_mean = (mean_t - mean_c) / mean_c * 100  # 227.8%

cap = np.percentile(control, 99)              # 20.0
treat_wins = np.clip(treatment, None, cap)
lift_winsorized = (treat_wins.mean() - mean_c) / mean_c * 100  # 0.7%

The mean shows a 227.8% lift in revenue per user (from 2.195 to 7.195), which looks like a spectacular win. The median lift is 0%: the typical user's behavior is completely unchanged. Winsorizing at the control arm's 99th percentile (capping treatment at 20) brings the "lift" down to 0.7%, essentially nothing. All three numbers come from the exact same underlying data; the only difference is one added $5,000 purchase in a sample of 1,000 users. Reporting the mean alone would tell a story about broad-based product improvement that the data doesn't actually support.

Trade-offs & pitfalls

• Don't pick whichever statistic tells the story you want. Report mean, median, and a robust variant together as standard practice, not as a reaction to a surprising result.
• Winsorizing/trimming thresholds should be set from a pre-specified rule (e.g. control-arm percentile), not tuned after seeing the treatment data, or you've reintroduced exactly the kind of post-hoc flexibility that inflates false positives.
• Log-transforms complicate interpretation for stakeholders; a "12% lift in log-revenue" needs translation into a business-meaningful statement (roughly a 12% multiplicative lift in revenue, not an additive one), and zero values break a plain log transform (hence $\log(1+x)$).
• If most of a real lift genuinely comes from a small number of high-value users, that's not automatically a false positive. It changes what to do next: consider whether that's the intended (e.g. a premium-tier feature) or unintended (e.g. one enterprise account skewing an SMB experiment) outcome, rather than assuming the whale should always be excluded.
• Segment-level Simpson's paradox checks matter as much as the tail-robustness fixes. A metric can be tail-robust and still misleading if the treatment changed who's in the denominator.

Direct answer

Solve the margin-of-error formula backward for $n$: given a target margin $E$, a confidence level (95%, so $z^{*}=1.96$), and an estimate of the standard deviation from a pilot sample, the required sample size is $n = (z^{*} s / E)^2$, rounded up. With the pilot's $s \approx 4$ and a target margin of 0.5, that comes out to about 246 customers, treated as a starting plan to be revisited once real data comes in.

Structured elaboration

Setting up the formula. A 95% CI for a mean has half-width (margin of error) $E = z^{*} \cdot SE = z^{*} \cdot \frac{s}{\sqrt n}$. Solving for $n$:

Using $z^{*}=1.96$ for planning (not $t$) is a standard simplification: at the sample sizes this formula tends to produce, $t$ and $z$ are close enough that using $z$ up front and refining with $t$ afterward is fine.

Why the pilot is only a starting point. The formula needs a standard deviation, but the true population $\sigma$ for customer lifetime value is unknown; that's exactly what the 40-customer pilot supplies as an estimate. Because it's an estimate from a small sample, it's noisy, and CLV in particular is often right-skewed with a long tail of high-value customers, which can make a 40-person pilot understate the true variance if none of the top-tail customers happened to land in it.

Worked example

Target margin $E = 0.5$, 95% confidence ($z^{*}=1.96$), pilot standard deviation $s = 4$ from $n_{pilot}=40$:

(Verified by direct computation.) If historical response/completion rates for this kind of data pull run around 80%, plan to sample or invite more than 246 to end up with 246 completed observations: $246 / 0.8 = 307.5 \Rightarrow 308$ invites.

Iterative refinement in practice.

1. Collect the initial planned batch (or an early tranche of it).
2. Recompute the sample standard deviation from the larger, more reliable batch. If it's meaningfully different from the pilot's 4, recompute $n$ with the updated $s$; a larger true $s$ means the original plan undershoots the target margin.
3. Once close to the target $n$, switch the final margin-of-error check to the $t$-distribution with $df = n-1$ for the precise value; at $n \approx 246$, $t_{0.975,245} \approx 1.970$ versus $z=1.96$, a difference of about 0.01 in the critical value, small enough that it rarely changes the plan.
4. Most product sample-size calculations stop at step 3. Two further refinements apply only in specific cases: if the population of eligible customers is small relative to $n$ (e.g. a niche segment with only a few thousand total customers), apply a finite-population correction (a downward adjustment to $n$ that accounts for sampling a large fraction of a small, finite population rather than an effectively infinite one), which shrinks the required sample size.
5. If sampling is clustered (e.g. by region or cohort) rather than a simple random sample, inflate $n$ by a design effect, or DEFF (a multiplier that accounts for the extra correlation between units sampled from the same cluster, which reduces how much independent information the same number of clustered units carries compared to a true simple random sample), to account for the extra correlation clustering introduces.

Trade-offs & pitfalls

• Treating the pilot-based $n=246$ as final rather than a planning estimate is the main risk: if the true variance is higher than the pilot suggested (common with skewed CLV data and a small pilot), the study will land with a wider-than-intended interval unless the sample size gets revisited.
• Non-response and attrition are easy to forget until data collection is already underway; inflating for expected response rate up front avoids a late scramble to recruit more.
• The margin-of-error formula assumes simple random sampling; ignoring clustering or stratification in the actual collection design while using the unadjusted formula will understate the sample size actually needed.

Direct answer

Model each variant's conversion rate with a Binomial likelihood and a Beta prior, which gives a closed-form Beta posterior for each arm; from there, sample or integrate to get the posterior probability that the variant beats control, the full uplift distribution, and an expected financial impact, rather than a single p-value. Pre-specify stopping rules on those posterior quantities (not on gut feel about when the numbers "look good") so the flexibility of Bayesian monitoring doesn't become an accidental form of optional stopping.

Structured elaboration

Model specification

For arm $i \in \{C, V\}$ with $n_i$ trials and $x_i$ successes:

Conjugacy gives a closed-form posterior:

Prior choice

Always run a sensitivity check across at least two of these; report how much the conclusion depends on the choice.

Decision rules and stopping

Pre-specify before looking at live data:

• Efficacy: stop and ship if $P(p_V > p_C) \geq 0.975$ and the expected uplift clears a minimum business-relevant threshold $\delta$.
• Futility: stop with no rollout if $P(p_V > p_C) \leq 0.05$, or $P(\text{uplift} > \delta) \leq 0.10$.
• Maximum horizon: a hard cap on sample size or calendar time regardless of posterior state, to bound operational risk.

Pre-specifying these thresholds (rather than picking a plausible-sounding number after seeing the data) is what keeps continuous Bayesian monitoring honest - the posterior probability itself does not automatically protect against a false-discovery-inflating "peek until it looks good" pattern.

Presenting to stakeholders

Report: posterior mean rate per arm, 95% credible interval on the uplift, $P(\text{variant beats control})$, $P(\text{uplift exceeds the business-minimum threshold})$, and expected financial impact with its own interval. Pair a posterior density plot (control vs. variant) with a single uplift-distribution histogram, since stakeholders generally find "here's the full range of plausible uplift" more actionable than a bare probability.

Worked example

Control: 100 conversions in 1,000 trials. Variant: 128 conversions in 1,000 trials. Prior: $\text{Beta}(1,1)$ for both arms.

import numpy as np
rng = np.random.default_rng(42)
a_c, b_c = 1 + 100, 1 + 1000 - 100   # Beta(101, 901)
a_v, b_v = 1 + 128, 1 + 1000 - 128   # Beta(129, 873)

samples = 200_000
pc = rng.beta(a_c, b_c, size=samples)
pv = rng.beta(a_v, b_v, size=samples)

p_beats = (pv > pc).mean()                        # 0.97559
delta = pv - pc
delta_mean = delta.mean()                          # 0.02790 (2.79 points)
ci_lo, ci_hi = np.percentile(delta, [2.5, 97.5])    # (0.00014, 0.05585)

Result: $P(p_V > p_C) = 0.976$, mean uplift 2.79 percentage points, 95% credible interval [0.01, 5.58] points. To stakeholders: "There's a 97.6% chance the variant converts better than control. The most likely uplift is about 2.8 points, with a plausible range of essentially flat to nearly 5.6 points better; the interval only barely brushes zero, so this comfortably clears a 0.975 efficacy threshold."

Translating that uplift into expected financial impact needs one more input: a value-per-conversion and a traffic volume. Suppose this funnel step sees 50,000 visitors a month and the average order value is $40:

monthly_visitors = 50_000
aov = 40

# reusing the posterior results computed above
delta_mean, ci_lo, ci_hi = 0.02790, 0.00014, 0.05585

extra_conversions_mean = delta_mean * monthly_visitors   # 1395.0
extra_conversions_lo = ci_lo * monthly_visitors          # 7.0
extra_conversions_hi = ci_hi * monthly_visitors          # 2792.5

impact_mean = extra_conversions_mean * aov   # 55800.0
impact_lo = extra_conversions_lo * aov       # 280.0
impact_hi = extra_conversions_hi * aov       # 111700.0

To stakeholders: "The expected financial impact is about $55,800 per month in incremental revenue, with a 95% credible range of roughly $280 to $111,700 per month, mirroring how wide or narrow the uplift interval itself is."

Sensitivity check

Re-running with a skeptical prior centered at the observed control rate ($\text{Beta}(10, 90)$, effective sample size 100) pulls both posteriors toward 0.10 and changes the conclusion:

import numpy as np
rng = np.random.default_rng(42)
samples = 5_000_000
ac, bc = 10 + 100, 90 + 1000 - 100   # Beta(110, 990)
av, bv = 10 + 128, 90 + 1000 - 128   # Beta(138, 962)
pc = rng.beta(ac, bc, samples)
pv = rng.beta(av, bv, samples)
p_beats = (pv > pc).mean()          # 0.9708

Result: $P(p_V > p_C) = 0.971$ under the skeptical prior, versus 0.976 under the uniform prior. That is below the 0.975 efficacy bar this answer's own decision rule requires before shipping, so the pre-specified stopping rule would not trigger under this prior, even though it does under the uniform one. This is exactly the kind of prior-sensitivity flip the check exists to catch: 1,000 observations per arm is not enough to fully overwhelm a skeptical prior with effective sample size 100 when the true effect sits this close to the threshold, so the honest read is "not yet robust to prior choice," not "unchanged." Flag results like this to stakeholders explicitly, since it means the prior is still doing real work in the conclusion, and either more data or an agreed-upon prior needs to be settled before shipping.

Trade-offs & pitfalls

• Continuous posterior monitoring without pre-specified thresholds reintroduces optional stopping. The posterior probability itself is always "valid" at any instant, but repeatedly checking it and stopping the first time it crosses an ad hoc threshold inflates the effective false-positive rate just as surely as peeking at frequentist p-values does; the fix is pre-committing to the stopping rule, not the Bayesian framework alone.
• Prior sensitivity is the main line of scrutiny a Bayesian result will get. Always show at least an uninformative-vs-skeptical comparison; a conclusion that flips between them is not yet strong enough to act on.
• Point-probability thresholds ($P > 0.975$) ignore magnitude. A tiny, business-irrelevant uplift can still cross a probability threshold with enough traffic; always pair the probability with the uplift-magnitude threshold ($P(\text{uplift} > \delta)$), not the probability alone.
• Beta-Binomial assumes independent, stable conversion probability per arm. Delayed conversions, non-stationary traffic mix, or per-user repeat exposure violate the i.i.d. Binomial assumption and require a more elaborate model (e.g., time-to-event or hierarchical structure).

Direct answer

A 95% confidence interval is a range built from your sample using a procedure that, if repeated many times, would produce an interval containing the true, fixed, unknown parameter about 95% of the time. It is a statement about the reliability of the procedure across hypothetical repetitions, not a 95% probability that this one specific, already-computed interval contains the true value. The true value either is or isn't in this particular interval; there's no probability left to talk about once the interval is fixed.

Structured elaboration

The correct frequentist interpretation

Before data is collected, the interval's endpoints are random, since they depend on which sample happens to be drawn; the true parameter is a fixed, unknown constant. "95% confidence" describes the long-run behavior of the interval-producing procedure: across many hypothetical repeated samples, about 95% of the intervals it produces would bracket the true value, and about 5% would miss it. Once actual data is collected and one specific interval is computed, say [41.2, 48.7], that interval either contains the true value or it doesn't. The randomness was in the sampling and calculation, not in the now-fixed true parameter or the now-fixed interval.

The common misreading, and why it's wrong

"There's a 95% chance the true value is in this interval" sounds almost identical to the correct statement but treats the true parameter as the random thing with a 95% probability of being in some location, when in the frequentist framework the true parameter isn't random at all, only the sampling procedure is. This misreading is extremely common, surveys of both students and working researchers find majorities endorse it, precisely because the language is subtle: "confidence" was chosen specifically to avoid claiming "probability" about a fixed parameter, but in everyday usage the two words feel interchangeable.

Why the distinction matters practically

It matters most when someone tries to use a single computed interval to make a probabilistic claim. "There's a 95% chance the true conversion lift is between 2% and 6%" is not licensed by a frequentist confidence interval, that statement would require a Bayesian credible interval, which is a genuine probability statement about the parameter given a prior. Swapping one framework's language onto the other's number is a real, if common, error, not just pedantry; it can lead to overconfident decisions when the sampling assumptions behind the interval, random sampling, correct variance model, are shakier than they look.

Plain-language version for a non-statistician

"We can't know the exact true number, only estimate it from our sample. This range is our best estimate plus a margin that accounts for sampling luck: if we ran this same measurement over and over on different random samples, about 19 times out of 20 the range we'd compute would contain the real number. So this range is the set of values consistent with what we observed, not a guarantee, but a calibrated one."

Worked example

Suppose (illustrative numbers, chosen to demonstrate the interpretation, not derived from a specific dataset) an A/B test reports a 95% confidence interval for the lift in conversion rate of [1.2%, 5.8%], entirely above zero.

• Correct statement: using this measurement procedure, if the experiment were rerun many times, about 95% of the resulting intervals would contain the true lift; this particular interval happens to be [1.2%, 5.8%].
• Correct practical use: because the interval excludes 0, the data are inconsistent with "no effect" at the 5% significance level, equivalent to a two-sided test rejecting the null of zero lift at alpha = 0.05.
• Incorrect statement to avoid: "there's a 95% probability the true lift is between 1.2% and 5.8%," which treats the fixed true lift as the random quantity.
• Incorrect statement to avoid: "95% of future users will see a lift in this range," which confuses a confidence interval for the mean parameter with a prediction interval for individual outcomes, a different and typically much wider interval.

Trade-offs and pitfalls

• The CI-versus-hypothesis-test duality, excludes zero if and only if significant at that alpha, is genuinely useful, but leaning on it too heavily can obscure that the interval also communicates magnitude and precision, which a bare p-value doesn't. Report the interval, not just significant or not significant.
• A narrow interval is not automatically good news; it just means the estimate is precise. A narrow interval tightly around a practically meaningless effect size is still a practically meaningless result.
• All of this assumes the sampling and modeling assumptions behind the interval are valid, random or representative sampling, correctly specified variance. A confidence interval computed on a biased or non-random sample gives a precise-looking interval around the wrong quantity; precision is not the same as accuracy.
• When talking to non-technical stakeholders, resist compressing the explanation down to "95% chance it's in this range," that's exactly the misreading. Use the "if we repeated this, most of the ranges we'd get would contain the truth" framing instead, even though it takes a few more words.

Direct answer

Decide pooled versus Welch by testing whether the two sample variances are plausibly equal, using Levene's test, which is robust to non-normality, rather than assuming equal variances by default. If Levene's p-value is above 0.05, treat the variances as equal and use the pooled-variance t-test with n1 + n2 - 2 degrees of freedom; otherwise use Welch's t-test, which does not assume equal variances and uses the Welch-Satterthwaite approximation for degrees of freedom.

Approach

1. Run Levene's test on the two samples to check the equal-variance assumption.
2. If the assumption holds (p > 0.05): pooled t-test, pooled variance, degrees of freedom = n1 + n2 - 2.
3. Otherwise: Welch's t-test, separate variances in the standard error, Welch-Satterthwaite degrees of freedom.
4. Compute the two-sided p-value from the Student-t distribution at that degrees of freedom, and a 95% confidence interval for the mean difference using the same standard error and critical value.

import numpy as np
from scipy import stats


def two_sample_ttest_with_ci(x, y, alpha=0.05):
    x = np.asarray(x, dtype=float)
    y = np.asarray(y, dtype=float)
    n1, n2 = len(x), len(y)
    if n1 < 2 or n2 < 2:
        raise ValueError("Each sample must have at least two observations.")

    # Levene's test decides pooled vs Welch (robust to non-normal data,
    # unlike Bartlett's test which assumes normality)
    stat_var, p_var = stats.levene(x, y)

    mu1, mu2 = x.mean(), y.mean()
    diff = mu1 - mu2
    s1_sq = x.var(ddof=1)
    s2_sq = y.var(ddof=1)

    if p_var > 0.05:
        test_type = "pooled"
        sp_sq = ((n1 - 1) * s1_sq + (n2 - 1) * s2_sq) / (n1 + n2 - 2)
        se = np.sqrt(sp_sq * (1 / n1 + 1 / n2))
        df = n1 + n2 - 2
    else:
        test_type = "welch"
        se = np.sqrt(s1_sq / n1 + s2_sq / n2)
        num = (s1_sq / n1 + s2_sq / n2) ** 2
        den = (s1_sq ** 2) / ((n1 ** 2) * (n1 - 1)) + (s2_sq ** 2) / ((n2 ** 2) * (n2 - 1))
        df = num / den  # Welch-Satterthwaite

    t_stat = diff / se
    p_value = 2 * stats.t.sf(np.abs(t_stat), df)
    t_crit = stats.t.ppf(1 - alpha / 2, df)
    ci = (diff - t_crit * se, diff + t_crit * se)

    return {
        "test_type": test_type,
        "t_statistic": float(t_stat),
        "degrees_of_freedom": float(df),
        "p_value": float(p_value),
        "confidence_interval_95": (float(ci[0]), float(ci[1])),
        "variance_test_pvalue": float(p_var),
    }

Key points

• Levene's test on the inputs, not an eyeball comparison of sample variances, drives the branch. That keeps the decision reproducible and automatable.
• Welch is the safer unconditional default in practice (it is what R's t.test() uses unless asked otherwise). This function only chooses pooled when Levene's test gives no evidence against equal variances.
• The confidence interval reuses the exact same standard error and degrees of freedom as the hypothesis test, so the CI and the p-value are always consistent: the CI excludes 0 exactly when p < alpha.

Worked example

Pinned seed np.random.default_rng(42), x drawn from Normal(52, 4) with n=30, y drawn from Normal(49, 9) with n=25:

Because y's variance (48.82) is roughly 5 times x's variance (9.65), Levene's test correctly rejects equal variances (p = 0.0018), the function selects Welch, and the fractional Welch-Satterthwaite degrees of freedom (31.85, not the pooled 53) reflects that the smaller, more variable sample limits how much the variance estimate can be trusted. Cross-checked against scipy.stats.ttest_ind(x, y, equal_var=False): t = 1.3823, p = 0.1765, matching the hand-rolled formulas.

Complexity

O(n1 + n2) time to compute means and variances, O(1) additional space beyond the inputs. Levene's test itself is O(n1 + n2).

Edge cases

• n1 < 2 or n2 < 2 raises, since sample variance is undefined.
• Zero-variance samples (all identical values) make Levene's test degenerate and can collapse the pooled-branch standard error toward 0, producing an enormous or undefined t-statistic. Guard this explicitly in production code.
• Heavy-tailed or strongly skewed data violate the normal-sampling-distribution assumption behind the t-distribution regardless of which branch is chosen. A bootstrap confidence interval or a nonparametric test such as Mann-Whitney U is the safer tool then, not a variance-equality fix.