Conversion Funnel Optimization Questions

Direct answer

Design the checkout-abandonment test around a single primary metric (checkout completion rate) with named guardrails (refund rate, payment failure rate, revenue per visitor), a sample size sized off the primary metric's baseline rate and minimum detectable effect, a measurement window long enough to cover a full weekly cycle plus any post-purchase guardrail lag, deterministic-hash bucketing so users are consistently assigned and isolated from other concurrent experiments, and a staged rollout (5% then 50% then 100%) gated on both the primary metric and every guardrail staying healthy at each stage.

Structured elaboration

Getting to the hypothesis before designing the test. A test does not start from "let's A/B test the checkout button"; it starts from a diagnosed problem: pull the funnel drop-off data for the checkout step, segment it (device, payment method, new versus returning), and find where drop-off concentrates. If mobile users abandon at 2x the desktop rate specifically at the payment-form step, the hypothesis becomes concrete and falsifiable, not "make checkout better" but a specific, testable causal claim, shown next.

Hypothesis. "Reducing the mobile checkout payment form from 3 required fields to 1 (with autofill) increases mobile checkout completion rate, because the current form's field count is the primary abandonment driver for mobile users at that step."

Primary and guardrail metrics. Primary: checkout completion rate (users who reach checkout_start and reach purchase, the metric the hypothesis is directly about). Guardrails, metrics that must NOT get worse even if the primary improves: refund/chargeback rate within 7 days of purchase, payment failure/decline rate, average order value, and page load time on the checkout step. Guardrails exist because a change can win on the primary metric while quietly damaging something else the primary metric cannot see; the concrete case worth naming explicitly: a simplified, lower-friction checkout can raise completion rate while ALSO raising refund and fraud rates, because some of the friction being removed (a required billing-address confirmation, for example) was doing real fraud-prevention or informed-consent work, not just slowing people down. A test that only looks at the primary metric would ship a change that looks like a clear win and is actually trading real revenue for a fraud or return-rate problem that shows up weeks later. Refund rate needs its own, LONGER measurement window than the primary metric for exactly this reason (see Measurement window below).

Sample size, approach (not full derivation). Sample size for a proportions test like this depends on four inputs: the baseline conversion rate (checkout completion today), the minimum detectable effect (the smallest lift worth being able to detect, decided by business judgment, not statistics, typically informed by the revenue value of that lift versus the cost of running the test longer), the desired significance level (commonly $\alpha = 0.05$), and the desired statistical power (commonly $1-\beta = 0.80$). Those four feed a standard two-proportion power calculation to produce the required sample size per arm; this answer names the inputs and the approach at a conceptual level, since the point here is knowing which four factors determine sample size and how they trade off against each other, not re-deriving the closed-form formula step by step. Two practical notes beyond the inputs: (1) the baseline rate should come from RECENT data for the exact segment being tested (mobile checkout specifically, not the blended overall rate), since a mismatched baseline silently under- or over-powers the test; (2) if traffic is limited, a smaller minimum detectable effect requires disproportionately more sample, so the MDE choice is often the practical lever that determines whether the test is even feasible in a reasonable timeframe.

Measurement window. Primary metric: at least one full week (to average out day-of-week effects like weekday-versus-weekend purchase behavior), and typically extended to cover 2 full weekly cycles for a stable read. Guardrails on a LAG, like refund rate, need a window that extends PAST the point where the last user entered the test, since a refund can happen up to the refund policy's window (7, 14, or 30 days) after purchase; declaring the test a winner from primary-metric data alone, before the guardrail's own lag window has closed, is how the conversion-versus-refunds tension above gets missed in practice.

Segmentation rules. Pre-register the segments you will check (mobile versus desktop, new versus returning, geography) BEFORE launching, not after seeing results; this is a bias-prevention rule, since hunting for a significant segment after the fact (the primary metric was flat, but "mobile users in Region X" moved) inflates the false-positive rate through unplanned multiple comparisons. If mobile is the hypothesized driver, mobile can be the PRIMARY analysis population rather than just a post-hoc segment, with desktop as a secondary check that the change did not regress the segment it was not designed for.

Rollout plan. Staged: 5% of eligible traffic for an initial health check (catch a broken build or an obviously bad guardrail signal fast, cheaply), then 50% once nothing looks wrong, then 100% once the test reaches its planned sample size and duration and the primary metric plus every guardrail clears their bar. Each stage gate is a genuine decision point, not a formality: a guardrail regression at 5% should halt the rollout before it reaches more users, not just get noted for later.

Worked example: novelty effects and personalization-decay

A test of a PERSONALIZED checkout flow (recommended payment method pre-selected based on past behavior) surfaces a pitfall generic to any personalization change, worth calling out explicitly rather than folding into the generic guardrail list above: a novelty effect can make week-1 results look better than the change's true steady-state impact, because some of the lift is users reacting to something new and different rather than to the personalization genuinely being more relevant. The concrete failure mode: a personalization model trained on stale behavioral data can also DECAY in relevance over time as user preferences drift, so a test that looks flat-to-positive in week 1 and week 2 can quietly erode by week 6 as the underlying model's recommendations grow less accurate relative to current behavior, an effect a short measurement window would never catch. The mitigation is to extend monitoring PAST the initial test window into a post-launch tracking period specifically watching for a downward trend in the personalized arm's lift over time, not just a single end-of-test readout, and to re-train or re-validate the personalization model on a defined cadence so a stale model is caught before its degraded relevance shows up as a silent conversion-rate regression.

Trade-offs and pitfalls: running many concurrent experiments (the experiment-platform question)

This question's own ask, how to handle users in multiple concurrent experiments and prevent cross-contamination, is a real system-design problem once a company runs more than a handful of tests at once, and deserves concrete technical treatment, not just a mention:

• Deterministic hash-based bucketing. Assign each user to a variant via a hash of (user_id, experiment_id) (for example hash(user_id + experiment_id) mod 100), NOT a random draw at request time. This gives two properties that matter: the SAME user always lands in the SAME variant across sessions and devices as long as user_id is stable (consistency), and, critically, hashing user_id combined with the experiment's OWN id means a user's bucket in experiment A is statistically INDEPENDENT of their bucket in experiment B, because the two hashes are effectively unrelated draws even for the same user. This independence is what allows many experiments to run simultaneously on overlapping populations without the assignments correlating with each other.
• Overlapping-audience management via layers/namespaces. A common pattern (used by large-scale experimentation platforms) organizes experiments into orthogonal LAYERS: within one layer, experiments are mutually exclusive (a user in the checkout-form-layout experiment cannot simultaneously be in a different checkout-form-layout experiment), while experiments in DIFFERENT layers (checkout layout versus a pricing-page experiment) can run concurrently on the same user, since they touch unrelated parts of the product and their effects are assumed roughly additive/independent. This lets a platform run dozens of tests at once without every pair needing manual conflict analysis.
• Explicit interaction checks for experiments that plausibly interact. Orthogonal hashing prevents ACCIDENTAL correlation, but it does not prevent two experiments from genuinely interacting in effect (a checkout-form simplification test and a pricing-display test both touching the same screen could interact even with independent bucketing). For experiments the team suspects might interact, either force them into the same layer (mutual exclusion) or run an explicit 2x2 factorial analysis on the overlap population to check whether the combined effect differs from the sum of the individual effects.
• Cross-contamination beyond assignment: leakage through shared infrastructure. Bucketing correctly is necessary but not sufficient; a shared cache, a shared recommendation model retrained on mixed-variant data, or a support agent manually "fixing" a treatment-group user's cart back to the control experience are all ways cross-contamination can happen AFTER correct assignment. Auditing for this means checking that variant-specific behavior does not leak into shared state that both arms read from.

Other pitfalls: running the test on a stage-X restatement (payment-entry drop-off, or onboarding/activation instead of checkout) uses this exact same framework unchanged, only the hypothesis, primary metric, and guardrails swap to match the new stage, nothing about the sample-size approach, bucketing, or rollout staging logic changes. Common mistake: treating the 5% rollout stage as a formality rather than a real gate, shipping straight to 50% because the initial numbers "look fine" after only a few hours, well before the guardrails (especially the lagged refund-rate guardrail) have had time to produce a meaningful read.

Direct answer

Applying RICE extended with an explicit strategic-importance term (reach, impact, confidence, effort, AND strategic importance, since the question names strategic importance as a required factor of the framework itself, not an optional afterthought) to the three ideas, using explicitly stated assumptions for the values the question does not give, ranks them A (simplify signup form) first by a wide margin, B (personalized onboarding emails) second, and C (new payment provider integration) last. The scoring formula is $\text{RICE-S} = (\text{Reach} \times \text{Impact} \times \text{Confidence} \times \text{Strategic}) / \text{Effort}$, and the ranking is driven less by the given impact multipliers alone than by reach and effort, A touches every user at almost no engineering cost, while C touches a narrower population at a much higher cost, which the impact multiplier alone does not capture; the strategic-importance term moves C's score up somewhat (it resolves a real open question about a whole payment-provider bet) without changing the overall order, since A and B's advantage on the other three factors is too large for a single strategic multiplier to overcome.

Structured elaboration

Why RICE, and what it needs beyond what the question gives. The question supplies only impact (as a multiplier) and a categorical effort label per idea; RICE also needs reach (how many users the change actually touches) and confidence (how sure the estimate is), neither of which is given, so a defensible answer states its assumptions explicitly rather than silently inventing numbers.

Stated assumptions, used consistently across all three ideas:

• Reach, in users per month, against an assumed top-of-funnel volume of 10,000 signups per month: A (simplify signup form) touches 100% of that volume, since every signup hits the form, giving reach 10,000. B (personalized onboarding emails) touches only users who complete signup and enter onboarding, assumed at 70%, giving reach 7,000. C (new payment provider) touches only users who reach the payment step, assumed at 40%, giving reach 4,000.
• Confidence, on RICE's conventional high/medium/low ladder (here 90%/70%/50%), based on how well-understood each mechanism is: A gets 90% (simplifying a signup form is a well-trodden, low-uncertainty UX pattern), B gets 70% (personalization/content effects are noisier and harder to predict precisely), C gets 50% (a full payment-provider migration has many uncontrolled variables affecting how much of the stated 1.6x impact is actually realized).
• Effort, mapped from the given qualitative labels to person-weeks of engineering and design time: low = 1, medium = 3, high = 8, a standard, explicitly stated conversion so the categorical labels become comparable numbers.
• Impact uses the multiplier given in the question directly as the RICE impact score (2.0, 1.3, 1.6).
• Strategic importance, a multiplier on a 1.0 (neutral)-to-1.5 (high) scale reflecting alignment with a broader company priority beyond this quarter's conversion number: A and B both get 1.0 (genuine, valuable, but not resolving any larger open strategic question); C gets 1.5, since a payment-provider migration also resolves a standing uncertainty about whether limited payment options are actually capping conversion at all, a question the team cannot answer without running something in this space regardless of C's raw RICE rank, and may separately reduce processing fees or unlock new markets, value RICE's other four terms do not capture on their own.

Worked example

All arithmetic computed in Python, not by hand:

ideas = {
    "A_simplify_signup_form":            {"reach": 10000, "impact": 2.0, "confidence": 0.90, "effort": 1, "strategic": 1.0},
    "B_personalized_onboarding_emails":  {"reach": 7000,  "impact": 1.3, "confidence": 0.70, "effort": 3, "strategic": 1.0},
    "C_new_payment_provider":            {"reach": 4000,  "impact": 1.6, "confidence": 0.50, "effort": 8, "strategic": 1.5},
}
for name, v in ideas.items():
    score = (v["reach"] * v["impact"] * v["confidence"] * v["strategic"]) / v["effort"]
    print(f"{name}: RICE-S = ({v['reach']} * {v['impact']} * {v['confidence']} * {v['strategic']}) / {v['effort']} = {score:.2f}")

Output (actually executed):

A_simplify_signup_form: RICE-S = (10000 * 2.0 * 0.9 * 1.0) / 1 = 18000.00
B_personalized_onboarding_emails: RICE-S = (7000 * 1.3 * 0.7 * 1.0) / 3 = 2123.33
C_new_payment_provider: RICE-S = (4000 * 1.6 * 0.5 * 1.5) / 8 = 600.00

Ranking: 1. A (18,000), 2. B (2,123.33), 3. C (600). The gap between A and the other two is large enough that it is not sensitive to modest changes in the assumed reach, confidence, or strategic-importance figures; A wins primarily because it combines the highest reach, the highest impact multiplier, the highest confidence, AND the lowest effort, every factor pointing the same direction. B and C's relative order is closer and more assumption-dependent: C's higher raw impact multiplier (1.6x versus B's 1.3x) AND its strategic-importance boost (1.5x versus B's neutral 1.0x, raising C from 400 to 600) are still more than offset by its narrower reach, lower confidence, and much higher effort (8 person-weeks versus 3), which is exactly the kind of trade-off a raw impact-multiplier comparison alone would miss and RICE's effort, reach, and strategic terms are designed to surface.

A near-identical restatement worth noting briefly. The same approach applies directly to a version of this question with 12 candidate experiments and a custom weighted scoring formula instead of naming RICE or ICE (impact, confidence, ease) explicitly: the mechanics are identical, assign explicit numeric values to whatever factors the custom formula names, state the assumptions, compute each candidate's score, and rank, just with more candidates and a locally-defined formula in place of RICE's four specific factors; it does not require a second worked derivation here, since nothing about the underlying method changes with more candidates or a differently named formula.

A version that also asks for an executive recommendation. If asked to add a one-slide executive recommendation on top of the ranking above, the slide should lead with the recommendation (A first, on the basis of its dominant RICE-S score and its low execution risk) and explicitly weigh one more factor the score still does not fully capture: measurement cost (A is also the cheapest to measure conclusively, a signup-form change reaches full statistical power quickly given its high reach, while C's narrower, high-effort nature means both building AND validating it will take materially longer). A senior recommendation names this tension explicitly rather than treating the RICE-S ranking as a mechanical final answer: ship A first on its own merits, but flag C as a candidate to sequence deliberately once the calendar allows, precisely because its strategic-importance weighting (already folded into its 600 score above) says the team should not simply let it sink to the bottom of the backlog forever, even though it is not next.

Trade-offs and pitfalls

• RICE's output is only as defensible as its inputs; two people scoring the same three ideas with different but equally reasonable reach or confidence assumptions can produce a different B-versus-C order (though A's dominance across every factor here makes its first-place rank robust to that kind of disagreement), so state assumptions alongside any RICE ranking, not just the final scores.
• A common mistake is treating the impact multiplier given in a prompt like this as already comparable across ideas without checking what population it applies to; C's 1.6x is a larger raw multiplier than B's 1.3x, and reading the multipliers alone would rank C above B, which is exactly backwards once reach and effort are accounted for, showing why RICE's other three factors are not optional extras.
• Effort mapped to person-weeks is itself an estimate with its own uncertainty (a "high effort" payment-provider integration could run well past 8 person-weeks if it hits an unanticipated vendor integration issue), and a sensitivity check, recomputing the ranking with a plausible range of effort values rather than a single point estimate, is a stronger practice than presenting one number as certain.
• Using RICE's score as the sole and final word, rather than as one input alongside strategic considerations like learning value and measurement cost (as the executive-recommendation extension above shows), risks systematically deprioritizing legitimately important but harder-to-score bets in favor of whatever scores well mechanically.

Direct answer

With Step A (landing) = 100,000, Step B (signup) = 8,000, and Step C (paid) = 1,200: overall conversion A to C is 1.2%, step A to B is 8%, step B to C is 15%, drop-off at A to B is 92%, and drop-off at B to C is 85%. The A to B step is the priority: it has both the lower conversion rate and, by a wide margin, the larger absolute number of users lost.

Structured elaboration

Formulas. For any two steps, conversion rate is users at the later step divided by users at the earlier step, and drop-off is one minus that rate (equivalently, users lost divided by users at the earlier step):

Overall conversion across a multi-step funnel is either computed directly (final step users divided by first step users) or as the product of the individual step rates, and both must agree:

Applying the formulas:

• Overall A to C: $1{,}200 / 100{,}000 = 1.2\%$
• Step A to B: $8{,}000 / 100{,}000 = 8\%$
• Step B to C: $1{,}200 / 8{,}000 = 15\%$
• Drop-off A to B: $1 - 8\% = 92\%$, meaning 92,000 of the 100,000 landing-page visitors never signed up.
• Drop-off B to C: $1 - 15\% = 85\%$, meaning 6,800 of the 8,000 signups never paid.
• Check: $0.08 \times 0.15 = 0.012 = 1.2\%$, matching the direct overall calculation.

Which step to prioritize. Two lenses matter and they can disagree, so check both: the RELATIVE rate (which step converts a smaller share of the people who reach it) and the ABSOLUTE volume lost (which step, in raw user count, is bleeding the most people, since a small percentage improvement on a huge base can outweigh a large percentage improvement on a small base). Here they agree: A to B converts at only 8% (versus B to C's 15%), and A to B also loses 92,000 users in absolute terms versus B to C's 6,800, roughly 13.5 times more. Both signals point the same direction, so A to B (landing to signup) is the clear priority. When the two lenses disagree, in general prioritize by the absolute number of users recoverable at a plausible improvement size, since that is what actually shows up in the paid-conversion count at the bottom of the funnel, not the percentage alone.

Worked example

Uplift-projection variant: what if a step's conversion rate improved? Take a separate, illustrative scenario built on the same funnel shape and the same 100,000-visitor base: suppose the A to B step conversion rate improved from a baseline of 20% to 30% (an illustrative before/after pair, distinct from this funnel's actual 8% observed rate, used here to demonstrate how an improvement propagates downstream). Holding the actual B to C rate fixed at 15%:

• Before: $100{,}000 \times 20\% = 20{,}000$ reach step B; downstream paid conversions: $20{,}000 \times 15\% = 3{,}000$.
• After: $100{,}000 \times 30\% = 30{,}000$ reach step B; downstream paid conversions: $30{,}000 \times 15\% = 4{,}500$.
• Incremental paid conversions: $4{,}500 - 3{,}000 = 1{,}500$, a 50% relative increase in paid conversions from a 10-percentage-point improvement at the first step.

This illustrates the general propagation rule: an improvement at an EARLY step multiplies through every downstream rate, so the revenue impact of fixing the A to B step is proportionally larger than the same percentage-point improvement made at B to C, purely because more of the funnel's total user base passes through the earlier step.

A near-identical restatement worth naming, not reworking. A version of this question that swaps the funnel to a Discover, Activate, Retain framing with the same "which step would you prioritize" ask, or one that relabels the raw counts as page views instead of landing-page visits, is testing the exact same skill: compute each rate, compute drop-off, compare relative and absolute loss, and justify a prioritization call. Nothing in the underlying reasoning changes when only the stage labels or the cover story changes.

Trade-offs and pitfalls

• Percentage points versus relative percentage. Saying a rate "improved by 10%" is ambiguous: going from 20% to 30% is a 10-PERCENTAGE-POINT increase but a 50% RELATIVE increase. Always state which one you mean; conflating the two is a common way to overstate or understate an improvement's real size.
• A single point-in-time snapshot hides trend. These three numbers are a snapshot; without knowing whether 8% at A to B is stable, improving, or degrading over time, prioritizing purely off one month's numbers risks reacting to noise rather than a real pattern.
• Absolute volume can mislead when a step's population is not truly comparable. If Step A includes a large amount of bot or low-intent traffic that Step B naturally filters out, the 8% rate may partly reflect traffic quality rather than a fixable UX problem; a prioritization decision made purely on the arithmetic above should be sanity-checked against what is actually driving the low rate before committing resourcing to "fix" it.
• Common mistake: treating drop-off percentages as if they were independent of each other, and assuming fixing the worse-percentage step alone doubles overall conversion. Because overall conversion is the PRODUCT of step rates, an improvement at one step still gets multiplied by the (unchanged) rate of every other step, so the overall lift is smaller than the step-level percentage-point gain might suggest, exactly as the uplift-projection example above shows (a 10-point step improvement yielded a 50% relative lift in paid conversions, not a 10-point lift in the overall rate).

Direct answer

Treat a view-through impression as a real but structurally weaker touchpoint than a click: gate it on whether it was actually viewable (an unseen impression earns zero credit, full stop), then weight the surviving impressions with a time-decay function so a view three days before conversion counts for less than a click one day before conversion. The weighting scheme is a hypothesis, not ground truth, so it has to be checked against a randomized holdout experiment (a public service announcement, PSA, or "ghost ad" design) that measures the actual incremental lift a view causes, and recalibrated when the two disagree.

Structured elaboration

Viewability as a hard gate, not a weighting factor. An "impression" served to a page is not the same as an impression a person actually saw; standard viewability definitions require a minimum percentage of pixels on screen for a minimum continuous duration (commonly at least 50% of pixels visible for at least one second for display ads, longer for video). Any impression that never met that bar gets zero view-through credit, because there is no plausible mechanism by which an ad the user never saw could have influenced them. This is the single biggest lever against inflated view-through numbers: most of the value of "getting viewability right" comes from correctly zeroing out the impressions that should never have entered the credit pool at all, before any decay math runs.

Recency and time decay. Among viewable impressions, credit should decay with elapsed time between the impression and the conversion, reflecting a genuine belief: an ad seen three weeks ago is a weaker candidate explanation for today's purchase than one seen three hours ago. A standard functional form is exponential decay:

$w(t) = e^{-\lambda t}$

where $t$ is time elapsed between the touchpoint and the conversion, and $\lambda$ is a decay-rate parameter tuned per product (a short-consideration product like a food-delivery app should use a larger $\lambda$, meaning old touchpoints decay fast; a long-consideration product like enterprise software should use a smaller $\lambda$).

Combining view and click credit in one model. Clicks are a stronger behavioral signal than views (the user took an action), so a view's weight should be discounted relative to a click's weight at the same elapsed time, using a view-discount multiplier $\alpha \in (0, 1)$:

$w_{\text{view}}(t) = \alpha \cdot e^{-\lambda t} \qquad w_{\text{click}}(t) = e^{-\lambda t}$

$\alpha$ and $\lambda$ are both tunable parameters, not universal constants; the Worked example below picks illustrative values to show the arithmetic, and the Validate section below explains where the REAL values should come from.

Normalizing to conversion value. For one converting user's path, compute the raw weight for every surviving (viewable, within-window) touchpoint, sum them, and divide each touchpoint's raw weight by that sum to get a credit share; multiplying each share by the conversion's dollar value gives the per-channel revenue credit, and the shares are constructed to sum to exactly 1 so the full conversion value is always fully allocated, never over- or under-counted.

Validating the model experimentally. An attribution weighting scheme, however principled, is still an assumption about causality read off correlational data: a person who was going to buy anyway might also happen to have seen an ad, and a naive model would credit the ad for a conversion it did not cause. The standard way to get an unbiased estimate of a view's TRUE incremental effect is a randomized "ghost ad" or public-service-announcement (PSA) holdout: randomly assign users into a treatment group that sees the real ad and a control group that sees an equal-weight PSA or filler ad instead (so both groups have an identical ad-serving experience, controlling for anything correlated with simply being ad-eligible), then measure:

$\text{Incremental lift} = \text{conversion rate}_{\text{treatment}} - \text{conversion rate}_{\text{control}}$

This incremental lift is the unbiased benchmark. The observational, decay-weighted credit model is then checked against it: if the model's implied view-through contribution for that channel over the same period is far from the experimentally measured lift, the $\alpha$ and $\lambda$ parameters (or the viewability threshold) are recalibrated until the two are reasonably close. The observational model stays useful because it is cheap and runs continuously on every conversion; the experiment stays necessary because it is the only source of truth for whether the model's shape is even approximately right.

Worked example

One converting user, three touchpoints, conversion value $150. Illustrative parameters (not universal constants): decay rate $\lambda = 0.15$ per day, view-discount $\alpha = 0.4$.

Channel A is excluded entirely at the viewability gate before any decay math runs. For B and C:

$w_B = \alpha \cdot e^{-\lambda \cdot 3} = 0.4 \cdot e^{-0.45} \approx 0.4 \times 0.63763 \approx 0.25505$

$w_C = e^{-\lambda \cdot 1} = e^{-0.15} \approx 0.86071$

$\text{sum} = w_B + w_C \approx 0.25505 + 0.86071 = 1.11576$

Normalized shares:

$\text{share}_B = \frac{0.25505}{1.11576} \approx 0.2286 \qquad \text{share}_C = \frac{0.86071}{1.11576} \approx 0.7714$

Check: $0.2286 + 0.7714 = 1.0000$, so the shares fully partition the conversion.

Dollar credit at $150 conversion value: $150 \times 0.2286 \approx \$34.29$ to Channel B, $150 \times 0.7714 \approx \$115.71$ to Channel C, summing to $150.00 exactly. Channel C's click, despite being only two days more recent than Channel B's view, ends up with more than three times the credit, which is the intended behavior: recency compounds with the view-discount to make a click strongly dominate a nearby but non-clicked impression, while Channel A's non-viewable impression, ten days out, contributes nothing at all regardless of how the decay curve is tuned.

Trade-offs and pitfalls

• View-through credit is easy to game if the window and cap are not bounded. Because a view requires no user action, a campaign can serve enormous impression volumes to anyone who ever visited the site and claim credit for any of them who eventually convert, even months later, if the attribution window is left open-ended. Bound the view-through window explicitly (commonly much shorter than the click-through window) and cap how many view-through credits a single user's conversion can receive from repeated impressions of the same ad, so frequency alone cannot inflate a channel's apparent contribution.
• The PSA/ghost-ad experiment is the right benchmark but is not free to run continuously. It requires holding back real ad delivery from a control group, which has a real opportunity cost, so most organizations run it periodically (a calibration exercise) rather than as an always-on measurement, then let the cheaper observational model carry the day-to-day reporting between calibrations.
• Common mistake: applying the same decay rate to every channel and product. A display retargeting impression and a connected-TV impression have very different plausible windows of influence; forcing one global $\lambda$ either overcredits the slower-influence channel or undercredits the faster one.
• Common mistake: conflating "viewable" with "influential." Meeting the viewability threshold proves the ad was on screen, not that the user registered it; viewability is a necessary floor for including a touchpoint at all, not evidence on its own that the touchpoint deserves a high weight, which is exactly why the decay-and-discount weighting still applies on top of the viewability gate rather than replacing it.

Direct answer

Moving free-to-paid conversion from 2% to 5% is a 150% relative increase, too large to bet on any single test, so the right deliverable is a sequenced, 6-month roadmap of experiments, front-loaded with the highest-confidence, lowest-effort changes to build early evidence and momentum, escalating toward bigger, riskier swings once the cheap wins are exhausted. Every experiment in the sequence needs its own stated hypothesis, primary metric, and success threshold, and the roadmap as a whole needs an honest, explicit check on whether the cumulative target is actually achievable in the stated window, which is a distinct and necessary part of the answer, not just the list of experiments itself.

Structured elaboration

Why sequencing, not just a list, matters. Running experiments in priority order rather than in parallel or arbitrary order does two things a flat list does not: it lets each experiment's real result inform whether the next one's underlying assumption still holds (an onboarding-friction fix that under-performs suggests the team's theory of what's blocking conversion needs revision before investing in a bigger, related bet), and it produces early, credible wins that justify continued investment in the later, more expensive experiments to skeptical stakeholders.

Sequencing logic used below. Early months prioritize changes that are cheap to build, fast to measure, and target a well-understood friction point (signup and first-session experience) where a positive result is likely and the downside of a negative result is small. Later months escalate to changes that require more engineering investment, take longer to reach a valid read (a trial-length experiment cannot conclude faster than the trial itself), or touch acquisition mix rather than only in-product behavior. This is the same confidence-versus-effort logic behind formal prioritization frameworks like RICE (reach, impact, confidence, effort) or ICE (impact, confidence, ease); here that logic is applied directly to produce an ordering, not re-derived as its own separate framework.

Alternative ways to arrive at a similar sequence, worth naming briefly. The same prioritized-experiment-sequence approach applies directly if the target metric were a B2B activation rate moving from 20% to 35% instead of a freemium-to-paid rate moving from 2% to 5%: the sequencing LOGIC (cheap and high-confidence first, expensive and uncertain later) does not change, only the specific experiments shift toward B2B-relevant activation levers (a guided admin setup wizard, a dedicated onboarding call for larger accounts) instead of freemium in-product prompts. A second, equally valid way to derive a sequence is customer-journey mapping: lay out every touchpoint from signup through billing (signup, first login, first key action, the upgrade moment, checkout) and design one experiment per touchpoint, rather than ranking by confidence and effort; this tends to produce a similar set of experiments organized by WHERE in the journey they sit rather than by WHEN to run them, which is a useful alternate lens when communicating the roadmap to stakeholders who think in journey stages.

Worked example

The month-6 sample-size calculation, worked explicitly (compact version: applied directly below using the standard two-proportion formula, without re-deriving it from first principles). Using the standard two-proportion sample-size formula

with $p_1 = 0.08$, $p_2 = 0.11$, $z_{\alpha/2} = 1.9600$ (5% two-sided significance), and $z_{\beta} = 0.8416$ (80% power), the computed sample size is 1,499 users per arm (2,998 total), actually computed in Python rather than estimated:

import math
from statistics import NormalDist
nd = NormalDist()
p1, p2, alpha, power = 0.08, 0.11, 0.05, 0.8
z_alpha, z_beta = nd.inv_cdf(1 - alpha/2), nd.inv_cdf(power)
p_bar = (p1 + p2) / 2
n = ((z_alpha*math.sqrt(2*p_bar*(1-p_bar)) + z_beta*math.sqrt(p1*(1-p1)+p2*(1-p2)))**2) / (p2-p1)**2
print(f"n per arm (raw) = {n:.2f}, ceil = {math.ceil(n)}, total = {2*math.ceil(n)}")

Output (actually executed): n per arm (raw) = 1498.39, ceil = 1499, total = 2998. This is what actually determines whether month 6 is even a realistically completable experiment: if this segment has materially fewer than roughly 3,000 eligible new users across the month, the experiment as designed cannot reach a conclusive read in the available time, and either the effect size being tested for needs to be larger (accepting the risk of missing a real but smaller effect) or the experiment needs to run longer than the single month slotted for it in this roadmap.

Trade-offs and pitfalls

• The arithmetic reality of the target deserves explicit acknowledgment. Moving from 2% to 5% free-to-paid conversion is a 150% relative increase; even optimistic individual experiments in a reasonably mature product typically produce single-digit-to-low-teens percent RELATIVE lifts each, not 150% in aggregate from six experiments. A senior answer states this tension directly rather than presenting the roadmap as guaranteed to hit the target: either the current 2% baseline reflects specific, large, fixable frictions (plausible if the product is young or has known major UX gaps, in which case bigger individual lifts are genuinely achievable), or the 5% target itself needs to be revisited as the roadmap's early results come in, with the sequencing designed to surface that reality quickly (front-loading experiments) rather than only in month 6.
• Each experiment's success threshold should be tied to what the CUMULATIVE roadmap needs to add up to the target, not chosen independently per experiment; a set of individually "successful" experiments that each clear their own small threshold can still fall well short of the 2%-to-5% goal if nobody checked the arithmetic adds up across all six.
• A common mistake is running all six experiments as fully independent, non-interacting bets; several of these (the upgrade prompt, the personalized emails, the paywall redesign) touch overlapping parts of the same conversion moment and can interact (a redesigned paywall changes the context the upgrade prompt is shown in), so later experiments in the sequence should account for the compounding effect of earlier shipped changes rather than being designed and analyzed in isolation as if the product were unchanged.
• The app-store-listing experiment (month 5) is measurement-different from the others in a way worth flagging: because it changes acquisition mix rather than in-product behavior, its "success" metric is necessarily cohort-based (comparing users acquired before versus after the change) rather than a randomized A/B split of existing users, which is a weaker causal design and more vulnerable to confounds like seasonality; treat its result with correspondingly more caution than the randomized in-product experiments.