About and methodology

Conversion rate

CR = conversions / users

Relative and absolute lift

lift = (CRv − CRc) / CRc; Δ = CRv − CRc

Effective alpha

α′ = (1 − confidence) / (groups − 1)

With Bonferroni and 3+ groups. Without correction, α′ = 1 − confidence. Bayesian analysis ignores Bonferroni and hypothesis.

Critical values and power

zα = Φ⁻¹(1 − α′/2) [two-sided]; zα = Φ⁻¹(1 − α′) [one-sided]; zβ = Φ⁻¹(power)

Sample per variant

n = (zα√(2p̄(1−p̄)) + zβ√(p₁(1−p₁)+p₂(1−p₂)))² / (p₂−p₁)²; p₂ = p₁(1+lift); p̄ = (p₁+p₂)/2

Traffic, MDE and duration

p = weekly conversions / weekly traffic; n(w) = weekly traffic × traffic share × w / groups; days = ceil(n × groups / (weekly traffic / 7 × traffic share))

MDE is the lift requiring n(w) visitors, solved by bisection in the sample formula. Absolute MDE = p × relative MDE. Zero conversions prevent planning. p₂ cannot exceed 1.

Unpooled standard error

SE = √(CR(1−CR)/n); SEdiff = √(SEc²+SEv²)

Z test

z = (CRv−CRc)/SEdiff; p₂ = 2(1−Φ(|z|)); p₁ = 1−Φ(z); significant if p < α′

Confidence interval and observed power

CIΔ = Δ ± Φ⁻¹(1−α′/2)SEdiff; CIlift ≈ CIΔ/CRc; observed power = Φ(|z|−zα)

One-sided analysis shows only the lower bound, Δ − zα SEdiff. Relative CI is approximate. Observed power does not replace planning.

Bayesian model

prior = Beta(1,1); posterior = Beta(1+conversions,1+users−conversions)

50,000 Monte Carlo draws per group. Seed derived from users and conversions, giving reproducible results.

Probabilities, loss and credibility

P(beat) = P(CRv>CRc); P(best) = P(CRg = max CR); loss = E[max(CRc−CRv,0)]; relative loss = loss/CRc; interval = lift quantiles [(1−conf)/2,1−(1−conf)/2]

Sequential testing

g(α,t) = 2−2Φ(Φ⁻¹(1−α/2)/√t); t = current n / maximum n

Lan-DeMets O'Brien-Fleming type. One-sided g(α,t). Two-sided, each side g(α/2,t), using the gsDesign/ldbounds convention. Boundaries use Armitage-McPherson-Rowe recursive integration. Bonferroni precedes spending. Fixed sample grows by the design inflation factor. Without a plan, information fractions are equally spaced. Reaching maximum sample implies final analysis. No futility boundary.

Verdict

Bayesian: winner P≥conf; loser P≤1−conf. Z test: significant and Δ>0 wins; significant and Δ<0 loses only two-sided.

Without a conclusive result, achieved MDE ≤ reference lift means no meaningful difference. Above it means underpowered. In sequential mode, significant means crossing the boundary.

Additional days

max(0,ceil((required n − min group n) × groups / (total users / days)))

SRM

χ² = Σ(observed−expected)²/expected; expected = total users × split; df = groups−1; alert p<0.01

Revenue and forecast

monthly visitors = (total users/days) × 30/traffic share; extra conversions = monthly visitors × Δ; monthly contribution = extras × value; impact = contribution × months; forecast = monthly visitors × CRg × value × months

Currency changes labels and formatting only, with no exchange conversion. Both methods use observed rates. Inconclusive values are unconfirmed estimates.