A/B Test Design & Statistical Rigor Questions
Designing and statistically defending a controlled online experiment: framing a testable hypothesis, defining control and treatment variants, choosing the randomization unit, setting the primary success metric, and computing sample size, power, and minimum detectable effect. Covers the statistical foundations that make a readout trustworthy, including hypothesis testing, p-values, confidence intervals, statistical vs practical significance, and Type I/II error. Emphasizes avoiding the common pitfalls that invalidate a test, such as peeking, multiple-comparison inflation, underpowered designs, and how test duration and stopping rules affect the validity of conclusions.
You are evaluating a price increase (for example, raising a marketplace take rate or introducing a new fee) in a two-sided marketplace with network effects between buyers and sellers. Design an experiment that accounts for spillovers between the two sides: specify the randomization scheme, including whether to randomize by buyer, seller, or a shared cluster, how you would detect and quantify cross-side externalities, and what analysis approach you would use to estimate the long-run revenue impact under these network effects.
Sample Answer
Direct answer
The key design choice is randomizing at a unit large enough to contain the cross-side spillover a price or fee change creates: typically a market cluster (a geography, city, or self-contained supply region) rather than individual buyers or individual sellers, because a marketplace's two sides interact through the same local supply-demand pool. Within that, you deliberately vary treatment intensity across clusters (a partial-saturation or dose-response design) so you can separate the direct pricing effect from the cross-side externality it triggers, and you plan the revenue read as a staged measurement: an early operational window for the direct effect, followed by a longer holdout-based window because churn and re-equilibration on a two-sided market take longer to surface than a single-user metric would.
Structured elaboration
Why buyer-only or seller-only randomization fails here
If you randomize individual sellers into a higher take rate, a treated seller may raise prices, list less, or churn; buyers who would have transacted with that seller instead transact with a control-arm seller in the same market. The buyer side is now indirectly treated regardless of which arm assigned it, which is the same SUTVA-style interference problem as a social feed, except the shared medium is the local marketplace instead of a social graph. Randomizing individual buyers has the mirror-image problem on the seller side. Either choice contaminates the arm you intended to leave clean.
Randomization scheme
- Unit: market cluster (geography, city, or another boundary where most matching happens locally, e.g., delivery radius). This keeps most buyer-seller matching internal to a single treatment condition.
- Design shape: partial saturation. Instead of a flat 50/50 split, assign clusters to a small number of treatment intensities (for example, no increase, a modest increase, a larger increase) rather than a single on/off arm. This lets you trace how the cross-side response scales with the size of the change, which a single treatment level cannot distinguish from a fixed step change.
- Randomize at the cluster level, stratified on baseline liquidity (existing buyer-to-seller ratio, transaction volume) so clusters that already look structurally different are balanced across intensities before you start, reducing the chance that a treatment-intensity effect is confounded with a pre-existing market difference.
Detecting and quantifying the cross-side externality
- Because clusters are internally exposed to one intensity, you can compare a directly-treated side's metric (e.g., seller take-rate exposure) against the other side's metric within the same cluster (buyer conversion, buyer price sensitivity) to see if the fee change on sellers moved buyer-side behavior, and by how much, as intensity increases.
- The partial-saturation design turns this into a dose-response check: plot the buyer-side metric against assigned intensity across clusters. A flat line across intensities is evidence of a contained direct effect; a sloped line is direct, in-cluster measurement of the spillover, not an assumption about its existence.
- Compare a cluster's realized outcome to adjacent, untreated clusters it plausibly shares supply with (e.g., neighboring cities where sellers can relist), which is a direct check for leakage across the cluster boundary itself, not just across sides within a cluster.
Estimating the long-run revenue impact
- Short-run direct revenue (immediate take-rate math: transactions times the new fee) is mechanical and available immediately, but it is not the number that matters, because it ignores behavioral response.
- The number that matters is the net of three components measured over a longer window: the direct fee revenue, minus revenue lost to seller churn or delisting, minus revenue lost to buyer-side friction from any resulting price or availability change. Each of these three needs its own time horizon: fee revenue is immediate, seller churn plays out over weeks as sellers decide whether to stay, and buyer-side effects play out over the buyer's own return cadence.
- Because of that lag structure, hold a subset of clusters as a long-run holdout past the point where you make the initial ship decision. This is what lets you catch delayed seller attrition or buyer defection that would not have shown up in an early readout, and it is a standard practice for any monetization change with a plausible slow-churn tail, not something specific to marketplaces.
Worked example
A delivery marketplace tests a take-rate increase across 40 city clusters, split into two treatment intensities plus control (roughly 13-14 clusters each, stratified on baseline order volume so the three groups start with comparable liquidity). At the direct level, transaction-weighted take-rate revenue rises with intensity, as expected mechanically. The diagnostic step is checking buyer-side order volume within the same clusters: if buyer order volume also declines with intensity (a negative slope across the three intensity levels, measured, not assumed), that decline is the in-cluster, dose-response evidence of the cross-side externality: sellers responded to the higher take rate by raising prices or delisting, and buyers responded to that. The ship decision then nets the two effects (higher unit take rate, lower volume) into an actual revenue trajectory rather than trusting the mechanical fee-revenue number alone.
Trade-offs and pitfalls
- Cluster randomization costs statistical power relative to individual-level randomization, because the effective sample size is the number of clusters, which is typically far smaller than the number of users; this needs a longer test or fewer, larger clusters, and it is a real cost you should state up front rather than discover after the fact.
- A short observation window will understate the true cost of the change, because seller churn and buyer defection both lag the price change; shipping on the early direct-revenue number alone is the single most common mistake in this design.
- Neighboring-cluster leakage (a seller in a treated city relisting in an adjacent control city) is a real risk for marketplaces with mobile supply; check it explicitly rather than assuming cluster boundaries are airtight.
- Resist reaching for a full structural or instrumental-variable model as the default; those are appropriate when randomization is genuinely unavailable, but the partial-saturation cluster design above gives a directly measured effect and should be preferred whenever you can actually randomize.
Explain CUPED (Controlled Experiments Using Pre-Experiment Data) as a variance-reduction technique for A/B tests. Describe what pre-experiment data it requires, the assumptions it relies on, and in plain terms how the adjustment is computed. What makes a pre-experiment covariate a good or a poor choice for CUPED, and what goes wrong if you pick a poor one?
Sample Answer
Direct answer
CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance-reduction technique, published by Microsoft researchers (Deng, Xu, Kohavi and Walker, 2013), that uses a metric measured before the experiment started to strip out some of the between-user noise in the outcome metric, without touching the unbiasedness of the treatment-effect estimate. It requires one thing: a pre-experiment value of a covariate, ideally the same metric you are about to test, measured for the same units, from a period before assignment happened (so it cannot itself have been affected by treatment). The payoff is a tighter estimate of the same treatment effect from the same traffic, which functionally means you can detect the same effect size with less traffic or in less time.
Structured elaboration
What it requires and assumes
- A pre-experiment covariate X for every unit, measured strictly before random assignment (so treatment cannot have influenced it) and ideally correlated with the outcome Y you're testing.
- The scaling coefficient must be estimated without using the treatment label, typically pooled across both arms or from control alone. This is what keeps the adjustment from injecting bias: you are not letting "knowledge of which arm someone is in" leak into how you weight their pre-period value.
How the adjustment is computed, in plain terms
For each unit, adjust its outcome by subtracting a scaled version of how far its pre-experiment covariate was from the population average:
YiCUPED=Yi−θ(Xi−Xˉ)
where the scaling coefficient is the population covariance of X and Y divided by the variance of X:
θ=Var(X)Cov(X,Y)
Intuitively: if a user's pre-period activity was above average, some of their post-period activity being above average too is "expected" and not informative about treatment, so it gets subtracted out. The treatment effect is then estimated as the difference in adjusted outcome means between arms, YˉtreatCUPED−YˉcontrolCUPED, which is algebraically the same as the raw mean difference minus θ times the pre-period mean difference between arms; since randomization makes that pre-period difference expected to be zero, the adjustment does not shift the expected value of the estimate, it only removes the part of each unit's variance that X already explained.
The resulting variance reduction has a clean form: if ρ is the correlation between the pre-experiment covariate and the outcome, the adjusted estimator's variance is (1−ρ2) times the unadjusted variance. A stronger pre-period predictor buys a bigger reduction; a weak one buys almost nothing.
What makes a covariate good or poor
| Property | Good covariate | Poor covariate |
|---|---|---|
| Timing | Strictly pre-experiment, cannot be touched by treatment | Measured after assignment, or partially overlapping the experiment window |
| Correlation with Y | Strong (ideally the same metric's own pre-period value) | Weak or unrelated |
| Availability | Present for essentially all units (e.g., existing users with history) | Missing for a large share of units (e.g., brand-new users with no pre-period) |
| Stability | A stable behavioral trait | Noisy, one-off pre-period value |
What goes wrong with a poor choice
- Using a post-treatment or treatment-adjacent value as X breaks the "cannot be affected by treatment" requirement and reintroduces bias: you would be conditioning on something treatment itself moved, which is a different and more serious problem than losing precision.
- Using a weakly correlated covariate costs almost nothing in bias but also buys almost nothing in variance reduction; the (1−ρ2) factor is close to 1.
- Using a covariate that is missing for a large share of units (commonly new users with no pre-period history) forces a choice: drop those units (losing sample and possibly biasing toward existing users) or impute a value (introducing a modeling assumption you did not need before).
Worked example
Suppose, for illustration, we assume a pre-experiment 7-day conversion rate correlates with the in-experiment conversion rate at ρ=0.6 for a given product (a hypothetical, stated input, not an empirical claim about any real product). The variance reduction from CUPED is then derived directly:
Var(Y)Var(YCUPED)=1−ρ2=1−0.62=1−0.36=0.64
So the adjusted estimator's variance is 64% of the unadjusted one, a 36% reduction, given only the stated assumption ρ=0.6. If ρ had instead been a weak 0.2, the reduction would be 1−0.04=0.96, i.e., only a 4% cut, illustrating why the strength of the pre-period correlation, not just its existence, determines whether CUPED is worth implementing for a given metric.
Trade-offs and pitfalls
- CUPED does not help metrics with no meaningful pre-experiment history, most obviously new-user-only experiments; for those, look at stratification or other covariate-adjustment techniques instead.
- Estimating θ using data that has already been split by arm (rather than pooled or control-only) risks contaminating the coefficient with treatment information; keep the estimation step blind to arm assignment.
- CUPED reduces variance, it does not change what the true effect is; do not present the tighter confidence interval as evidence the effect itself grew, only that you can now see it more precisely.
- CUPED composes with stratified randomization (apply the adjustment within strata), which is a reason to know both rather than treat them as competing choices.
Beyond the initial launch experiment, why would you keep a long-run holdout group even after a feature or a pricing algorithm change has fully shipped? Explain how you would decide the size of the holdout, how long to maintain it, and what you are trying to learn from it that the original launch experiment could not tell you. How would you communicate the cost of maintaining a holdout to stakeholders who want the new experience rolled out to everyone?
Sample Answer
Direct answer
A launch experiment tells you what happens over its own short window; it cannot tell you what happens after novelty fades, after users have had months to adjust their real behavior, or after a pricing algorithm has compounded across several billing cycles. A long-run holdout is a small slice of the eligible population that is deliberately kept on the old experience indefinitely, purely so you still have a counterfactual after everyone else has moved on. Once you roll out to 100%, that counterfactual disappears unless you built one in on purpose, so the holdout is not a nicety, it is the only way to keep answering "compared to what" after full rollout.
Structured elaboration
What the launch experiment structurally cannot tell you
- Novelty and primacy effects. A novelty effect is a short-term bump from users noticing and exploring something new, which fades as the new experience becomes routine. A primacy effect is the opposite pattern: a change that depresses behavior briefly while users re-learn a workflow, then recovers or improves as they adapt. A one- or two-week launch test mostly measures whichever of these dominates early, not the steady-state effect.
- Compounding and delayed effects. A pricing algorithm change or an onboarding flow can shift 90-day retention, churn, or lifetime value in ways that simply have not happened yet by the time the launch test ends. There is nothing to measure early because the outcome has not occurred.
- Post-launch drift. Once a feature is fully shipped, everything else in the product keeps changing around it (other launches, seasonality, market conditions). Without a live control, you cannot separate the feature's ongoing effect from all of that background drift.
Sizing the holdout
Size the holdout the same way you would size any two-arm comparison: pick the smallest long-run effect you would regret missing on your slowest-maturing primary metric, then run the sample-size calculation against your actual eligible population.
n=(p2−p1)2(z1−α/22pˉ(1−pˉ)+z1−βp1(1−p1)+p2(1−p2))2,pˉ=2p1+p2
This gives you the minimum number of users per arm; convert that into a required holdout percentage against your eligible population size (worked below). Round the resulting percentage up, both for attrition out of the holdout itself and because a holdout that is exactly borderline-powered on day one will be underpowered a year later as the population shifts.
How long to maintain it
Tie the minimum duration to the natural maturation window of the outcome you actually care about (a 90-day retention outcome needs at least one full 90-day window past stabilization, not one arbitrary calendar month). Beyond that minimum, treat "keep the holdout" as a decision revisited on a fixed cadence (for example, quarterly) rather than a permanent default:
- If two consecutive readout windows show a stable, well-understood effect, that is the trigger to either retire the holdout or shrink it to a smaller size that still clears the power bar above.
- If the effect is still moving or the product around the feature keeps changing, that is the trigger to keep the holdout at full size.
Surrogate metrics while waiting for the readout
Waiting 90 days for the primary outcome does not mean flying blind for 90 days. Track short-horizon metrics that historically correlate with the long-run outcome, such as week-1 activation or day-7 return rate as leading indicators for 90-day retention, and monitor them on a lightweight, non-primary basis. Two things matter about surrogate metrics: they are for early warning only ("this is trending in a worrying direction, look closer"), and they do not substitute for the long-run readout, because a surrogate can move without the outcome it is supposed to predict actually moving in the same direction once the novelty period ends.
Keeping the holdout uncontaminated
SUTVA, the stable unit treatment value assumption, is the assumption that one unit's outcome does not depend on another unit's treatment assignment. A long-run holdout only tells the truth if it holds: if held-out and treated users interact (shared households, marketplace two-sidedness, referral loops), the "control" group is partly experiencing the treatment through spillover and the comparison is biased. Keep the holdout cohort assignment stable and out of unrelated concurrent experiments on the same surface, and periodically re-check that its demographics still resemble the overall population (holdout users who disproportionately churn out over time silently change what the holdout represents).
Communicating the cost to stakeholders
Frame the holdout as a bounded cost with a defined trigger to shrink it, not an indefinite tax on the business:
- State the cost concretely: holdout size times the per-user value of the already-measured launch uplift times the time period, so a stakeholder sees an opportunity-cost number in the same units as their other decisions, not an abstract appeal to rigor.
- Pair that cost with what it buys: the ability to catch a long-run reversal (a change that looked good for two weeks but erodes retention over two quarters) before it has already happened to 100% of users.
- Offer a shrinking schedule tied to the review cadence above, so "keep a holdout forever" is never the actual proposal on the table.
Worked example
Suppose the primary long-run outcome is 90-day retention, currently at a 40% baseline, and the team wants to be able to detect a 1 percentage point absolute erosion (39% vs 40%) at α=0.05 two-sided, 80% power (z1−α/2=1.9600, z1−β=0.8416).
pˉ=20.40+0.39=0.395
z1−α/22pˉ(1−pˉ)=1.9600×2×0.395×0.605=1.9600×0.6910=1.3544
z1−βp1(1−p1)+p2(1−p2)=0.8416×0.40×0.60+0.39×0.61=0.8416×0.6931=0.5833
n=(0.01)2(1.3544+0.5833)2=0.00013.7511≈37,513 users per arm
Against a population of 2,000,000 monthly eligible users, a 1% holdout (20,000 users) falls short of this bar, a 2% holdout (40,000 users) clears it with a small margin, and a 3% holdout (60,000 users) clears it comfortably and leaves room for attrition. That is the actual decision: 2% is the honest minimum for this MDE, 3% is the safer operating choice, and anything above that is buying detection of a smaller effect than the team said it cared about, at a cost that keeps growing.
Trade-offs & pitfalls
- Conflating a holdout with a canary. A canary (see ramp and staged rollout) exists to catch acute, short-term harm during rollout. A holdout exists to measure a slow-moving counterfactual after rollout is complete. Sizing and duration logic for one does not transfer to the other; a 2% canary held for three days answers a completely different question than a 2% holdout held for two quarters.
- Letting the holdout go stale. A holdout that was correctly sized against last year's population and last year's MDE can silently become underpowered as traffic composition shifts; the sizing calculation is not a one-time exercise.
- Treating "permanent" as the default answer. The strongest version of this answer is a holdout with an explicit re-evaluation trigger, not an open-ended commitment that stakeholders correctly resent paying for indefinitely.
Your A/B test shows no overall lift, but a particular user segment, say mobile users, shows a statistically significant positive uplift. How would you validate whether this is a genuine heterogeneous treatment effect rather than a false positive from looking at many segments? What analyses would you run, and if you're not yet certain, what decision process would you use to decide whether to ship for that segment, run a confirmatory follow-up experiment, or abandon the finding?
Sample Answer
Direct answer
Treat a single surprising segment finding, mobile shows a significant lift while the overall test is flat, as a hypothesis to validate, not a result to act on. Work through data-integrity checks, a formal interaction test with a multiplicity correction (since this segment was very likely noticed after the fact rather than pre-specified), and a set of robustness checks; then use an explicit decision process that weighs the statistical uncertainty against the business value and cost of being wrong, rather than a pure significance threshold, to choose between shipping to that segment, running a confirmatory follow-up, or abandoning the finding.
Structured elaboration
Step 1: verify the data before trusting the effect
- Check assignment balance within mobile specifically: treatment and control counts, and balance on key covariates, within the mobile slice alone, not just in aggregate.
- Check for instrumentation differences: missing events, a different SDK version, or a different exposure window on mobile that could produce a spurious effect having nothing to do with the treatment.
- Check for timing issues: did the mobile rollout start at the same time as the rest of the experiment, and is there any cross-over where a user appears in both device buckets across the test window.
Step 2: test the interaction formally
Fit an interaction model rather than comparing the mobile-only conversion rate to the mobile-only control rate informally:
import statsmodels.formula.api as smf
df["treat"] = df["assignment"].map({"control": 0, "treatment": 1})
model = smf.logit("conversion ~ treat + mobile + treat:mobile + signup_channel", data=df).fit()
print(model.summary())
Illustrative output (a hypothetical summary row, not a real run) would show a coefficient, standard error, z-value, and p-value for each term; the row that matters most here is treat:mobile. A row reading something like treat:mobile coef = 0.18, p = 0.02, alongside a treat main-effect coefficient close to zero and non-significant, is the pattern that supports a genuine mobile-specific effect: the interaction term carries the real signal while the main treatment effect alone looks flat, consistent with the original observation that the overall test showed no lift. A significant coefficient on treat:mobile is what actually supports "the effect really differs by device," rather than the mobile-only point estimate on its own, which can look large purely from within-mobile noise.
Step 3: correct for multiplicity honestly
Ask directly whether mobile was a subgroup chosen before the test ran or one noticed afterward because it happened to look interesting. If it was not pre-specified, and in practice it usually was not when this kind of question comes up, apply a multiplicity correction appropriate to however many segments were actually eyeballed (even informally) before mobile stood out, or at minimum treat the raw p-value as an optimistic upper bound on how surprising this finding really is.
Step 4: check power on the mobile slice itself
Compute the sample size and event count within mobile alone and the confidence interval width on its effect estimate. A wide interval or a small mobile sample means the "significant" reading is fragile, and this matters even more when mobile is a genuinely small-traffic segment (a specific device class or platform with limited volume) rather than merely a smaller slice of a large population: in that case a confirmatory follow-up restricted to the same segment may take a long time to reach adequate power, or may never fully reach the same statistical bar as the overall test, which is itself part of the decision, not a reason to ignore the finding.
Step 5: robustness checks
- Look at related metrics (engagement, retention, complaint or refund rate) to see whether they move in a direction consistent with the primary metric's mobile-specific lift, or whether the primary metric is moving alone in a way that is harder to explain.
- Check whether the effect is stable over the test window or concentrated in a short burst of days.
- Check finer sub-slices of mobile (iOS versus Android, OS version) to rule out the effect actually being driven by one narrow slice within "mobile" rather than the device class as a whole.
- Re-run with alternative covariate adjustment and see whether the interaction coefficient is stable.
Worked example: the decision process
Rather than a bare "p < 0.05 so ship it" rule, weigh four inputs explicitly: how strong the statistical evidence is after the checks above, how large and reliable the resulting business value would be if the effect is real, how costly it is if the segment is shipped and the effect turns out not to be real, and how long a confirmatory follow-up on that segment alone would realistically take to reach adequate power given the segment's own traffic volume.
- Strong evidence, low cost of being wrong, fast to confirm: ship a small, reversible rollout to the segment while a confirmatory read continues, since the downside of being wrong is small and quickly detected.
- Moderate evidence, or the segment is small enough that a proper confirmatory test would take a long time to reach power: this is the case worth naming explicitly, since waiting for full statistical certainty may never be practical for a genuinely small segment. Here, the decision becomes an explicit risk-tolerance call: state the estimated cost of shipping on an unconfirmed finding versus the estimated cost of never acting on a real effect because the segment could never generate enough data to confirm it on its own, and make that trade-off visible to the decision-maker rather than deferring it to a p-value the segment may structurally never be able to produce.
- Weak evidence, or a moderate cost of being wrong: run a dedicated, pre-specified confirmatory experiment targeted at the segment before making any production change, treating the original finding purely as the hypothesis that justified the follow-up.
- Evidence disappears after the data-integrity and robustness checks: abandon the finding and document why, so the same slice does not get re-litigated the next time someone happens to look at it.
Trade-offs & pitfalls
- Treating an unadjusted subgroup p-value as decisive. The interaction test plus a multiplicity correction is what separates a real segment effect from one of several plausible slices that happened to look significant.
- Waiting indefinitely for a small segment to reach the same statistical bar as the overall test. For a genuinely low-traffic segment, that bar may not be reachable on a useful timeline; the decision framework needs to say what happens in that case rather than defaulting to inaction.
- Ignoring instrumentation as a candidate explanation. A device-specific logging or SDK difference is a mundane but common cause of an apparent segment effect and should be ruled out before any statistical machinery is trusted.
- Shipping on a single significant slice with no plan to re-check it. Even a reversible segment rollout should carry a defined follow-up read, not be treated as a closed decision the moment it ships.
You need to plan how long an experiment must run. Given daily unique visitors, the traffic allocation per variant, baseline conversion rate, desired minimum detectable effect, alpha, and power, show how to compute the required sample size per variant and then convert that into an expected number of days to run the test. State the assumptions and rounding choices you make along the way.
Sample Answer
Direct answer
Convert a sample-size target into a run duration in two steps: compute the required sample size per variant with the standard two-proportion test formula, then divide that by how much daily traffic actually lands in each variant (daily uniques times allocation), rounding up. The formula gives you a headcount; the traffic split is what turns a headcount into a calendar.
Structured elaboration
Step 1: the sample-size formula
For a two-sided test comparing baseline conversion p1 against a target conversion p2, at significance α and power 1−β:
n=(p2−p1)2(z1−α/22pˉ(1−pˉ)+z1−βp1(1−p1)+p2(1−p2))2,pˉ=2p1+p2
This is per-variant sample size; assumes a two-sided test with no sequential peeking (one look at the end) and independent, one-conversion-per-user data. Sequential monitoring and multiple-comparisons corrections change this number and are a separate design decision, not part of the raw duration estimate.
Step 2: convert daily traffic into per-variant daily volume
daily per variant=daily unique visitors×allocation share
Step 3: convert sample size into days
days=daily per variantn
Round the sample size up (never down, since undershooting trades away the power you asked for) and round the resulting day count up to a whole day. If the metric has meaningful weekday/weekend variation, round further up to a whole number of weeks so every arm sees the same mix of weekdays and weekends; that decision is a separate seasonality question with its own reasoning (see the seasonality-planning answer), not something the raw formula above accounts for.
Assumptions and rounding choices worth stating out loud
- Two-sided test with a single, pre-planned final look; no interim peeking.
- Users are independent and contribute one conversion event each; no repeated exposure double counting.
- All intermediate values kept to about five significant figures before rounding the final answer, and the final day count always rounds up, not to the nearest day.
- A buffer of roughly 10-20% is a common practical addition on top of the raw day count to absorb data loss (bot filtering, QA holds, instrumentation gaps); state it as a buffer, not as part of the statistical requirement.
- If daily traffic is volatile rather than a single stable number, use a conservative (lower) daily estimate for the duration calculation rather than the average, since under-running the test is far more costly than over-running it by a day or two.
Worked example
Inputs: 60,000 daily unique eligible visitors, 50/50 allocation, baseline conversion p1=3.0%, desired minimum detectable effect of 8% relative, α=0.05 two-sided (z1−α/2=1.9600), power 80% (z1−β=0.8416).
Absolute MDE: Δ=0.03×0.08=0.0024, so p2=0.0324, pˉ=0.0312.
z1−α/22pˉ(1−pˉ)=1.9600×2×0.0312×0.9688=1.9600×0.2459=0.4819
z1−βp1(1−p1)+p2(1−p2)=0.8416×0.03×0.97+0.0324×0.9676=0.8416×0.2459=0.2069
n=(0.0024)2(0.4819+0.2069)2=0.000005760.4744≈82,376 users per variant
Daily per variant: 60,000×0.5=30,000.
days=30,00082,376≈2.75→round up to 3 days minimum
Because this crosses a weekday/weekend boundary either way, the practical duration recommendation would be at least 7 days (one full week) rather than the bare 3-day statistical minimum, so weekday and weekend behavior are represented in the same proportion in both arms.
Trade-offs & pitfalls
- MDE sensitivity. The required sample size scales with the inverse square of the MDE, so halving the effect you want to detect roughly quadruples the required sample and the resulting duration; a stakeholder asking for a smaller MDE "just to be safe" is asking for a much longer test, not a marginally longer one.
- Binary-outcome formula does not transfer to continuous metrics. Revenue-per-user or time-on-task outcomes use the outcome's variance, not p(1−p), in the same general formula shape; plugging a conversion-rate formula into a continuous metric silently understates or overstates the required sample.
- Shortcutting the calendar rounding. A raw day count under 7 does not mean the test is safe to run for that literal number of days; the weekly-cycle rounding matters as much as the raw arithmetic and is a common place teams cut a real corner under launch pressure.
- Traffic volatility. A single "daily uniques" number hides day-to-day swings; a duration plan built on a lucky high-traffic day will run short in practice.
Unlock Full Question Bank
Get access to all 22 A/B Test Design & Statistical Rigor interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.