Pre-test
Plan your sample before looking at the result.
MDE and sample size
In a noisy room, you only hear a whisper if you get close. A shout carries across the room. The MDE, or minimum detectable effect, is the smallest effect your test can hear with the sample it has. With little traffic, the test only hears shouts. For example, an MDE of 8.4% at 80% power means that if the variant lifts conversion by 8.4% or more, the test has at least an 80% chance of detecting it. Smaller lifts can go unnoticed.
see details
The baseline rate is p = weekly conversions / weekly traffic. In week w, each group has n(w) = weekly traffic × % in test × w / groups visitors.
The sample per variant for a target lift is n = (zα × √(2 × p̄ × (1 − p̄)) + zβ × √(p1 × (1 − p1) + p2 × (1 − p2)))² / (p2 − p1)², with p1 = p, p2 = p × (1 + lift) and p̄ = (p1 + p2) / 2. The MDE for week w is the lift that requires exactly n(w) visitors in that formula. The calculator finds it by bisection, so the two pre-test modes are exact inverses of each other. The absolute MDE is p × relative MDE, in percentage points.
Why low traffic only detects large effects. The sample grows with the inverse square of the difference, 1 / (p2 − p1)². Detecting half the effect takes about 4 times as many users. Going the other way, the MDE shrinks with the square root of the sample, so running 4 times longer only halves the MDE. A low baseline rate makes it worse, because the same relative lift becomes a smaller absolute difference.
Duration is days = ⌈ n × groups / (weekly traffic / 7 × % in test) ⌉. Each extra variant adds to the total sample, and the Bonferroni correction adds to each group's sample. With planned looks, n is multiplied by the inflation factor of the sequential design. Round the duration up to full weeks.
When it misleads. The MDE is not the effect you should expect. Picking an optimistic target lift shortens the plan but leaves the test blind to realistic effects. In practice, few CRO tests show true lifts above 10%. When a test with a high MDE comes out significant, the observed lift tends to overstate the real effect. The calculation also assumes the baseline rate and the traffic hold steady during the test.
Two-sided or one-sided
A two-sided test is like a guard watching both doors of a house. A one-sided test watches only the front door. It spots anyone coming through that door sooner, but it never sees who comes in the back. In an A/B test, the front door is the variant raising conversion. The back door is the variant lowering it. A one-sided test never calls a variant a loser, however bad it is.
see details
Two-sided. The null hypothesis is CRv = CRc and the alternative is CRv ≠ CRc. The critical z is z(1 − α'/2), which is 1.96 at 95% confidence, and the p-value is 2 × (1 − Φ(|z|)). It detects gains and losses.
One-sided. The null hypothesis is CRv ≤ CRc and the alternative is CRv > CRc. The critical z is z(1 − α'), which is 1.645 at 95% confidence, and the p-value is 1 − Φ(z). With the same sample, a one-sided test detects smaller gains. For the same MDE, it needs about 20% fewer users.
A one-sided test does not detect losses. If the variant converts less, z goes negative and the p-value gets close to 1. The verdict becomes inconclusive, never loser. For the same reason, the one-sided confidence interval only has a lower bound.
When it misleads. The choice has to be made before you see the data. Picking the direction after seeing the result effectively doubles the false positive rate. Use a one-sided test only when a loss and a tie lead to the same decision, which is not to ship, and when nobody will need to know whether the variant made things worse. The Bayesian method does not use this choice.
Confidence and power
A fire alarm can fail in two ways. It can ring with no fire, or stay silent during one. Confidence controls the first mistake and power controls the second. Making the alarm stricter cuts false alarms but raises the risk of missing a small fire. To reduce both at once, the test needs more data. In your test, confidence is 95% and power is 80%. If the variant has the target lift, the chance that the test detects it is 80%, as long as it runs to the planned sample.
see details
A type I error is calling a difference that does not exist, a false positive. Its probability is alpha, α = 1 − confidence. With Bonferroni on and 3 or more groups, each comparison uses α' = (1 − confidence) / (groups − 1).
A type II error is missing a difference that does exist, a false negative. Its probability is β = 1 − power. Power always refers to a specific effect size. 80% power means an 80% chance of detecting the difference if the true lift equals the target lift. For smaller true lifts, power is lower.
Both errors compete for the same sample, which comes from n = (zα × √(2 × p̄ × (1 − p̄)) + zβ × √(p1 × (1 − p1) + p2 × (1 − p2)))² / (p2 − p1)². Here p1 is the baseline rate, p2 = p1 × (1 + lift), p̄ = (p1 + p2) / 2, zβ = z(power), and zα = z(1 − α'/2) two-sided or z(1 − α') one-sided. With a fixed sample, raising confidence lowers power. To keep both, the sample has to grow. At 80% power, going from 95% to 99% confidence takes about 50% more users. At 95% confidence, going from 80% to 90% power takes about 34% more.
In the Bayesian method, confidence is the P(beat) threshold for calling a winner. There is no power there, but the "underpowered" verdict uses the same sample rule as the Z test.
When it misleads. 95% confidence does not mean a 95% chance that the result is right. If few of the ideas you test really work, far more than 5% of declared winners can be false. Power is not the chance that the result is right either. It is a property of the plan, computed before the test. Observed power, Φ(|z| − critical z), is just the p-value written another way. It sits near 50% when the p-value is right at the threshold and says nothing about whether the sample was large enough. For that, compare the achieved MDE with the target lift.
Peeking and sequential testing
Imagine you could blow the final whistle the moment your team takes the lead. You would win games you would lose over the full 90 minutes. Checking a test every day and stopping at the first significant result works the same way. Every look is another chance for luck to pass as a real effect. A sequential test agrees in advance on how many times you will look and demands a much bigger lead to stop early.
see details
Why peeking inflates false positives. When there is no effect, each analysis has an alpha-sized chance of coming out significant by luck. Looking several times and stopping at the first significant result piles those chances up. At 95% two-sided with the usual threshold at every look, the overall false positive rate climbs to about 8% with 2 looks, 14% with 5 and 19% with 10. With endless looks, it heads toward 100%.
How the boundaries fix it. A sequential test spends alpha gradually, using the Lan-DeMets O'Brien-Fleming-type spending function. One-sided tests use α(t) = 2 − 2 × Φ(z(1 − α/2) / √t). Two-sided tests spend half of alpha on each side with the same function, α_side(t) = 2 − 2 × Φ(z(1 − α/4) / √t), the gsDesign and ldbounds convention. Here t is the information fraction, current n / maximum planned n. The boundaries come from that function by recursive numerical integration, the Armitage, McPherson and Rowe method. By the last look, the alpha spent adds up to exactly the alpha you chose. One-sided tests use the same idea with a single upper boundary. With Bonferroni on, the correction comes first and the corrected alpha is then spread across the looks.
The early boundaries are very strict, and the last one sits close to the usual critical z. With 4 equally spaced looks at 95% two-sided, the boundaries are about z = 4.33, 2.96, 2.36 and 2.01. In exchange, the maximum sample grows only a little. With 5 looks and 80% power, it is about 2.5% larger than the fixed-sample test.
The calculator uses the actual sample at each look to compute t, not the planned one. That keeps alpha correct if a look comes early or late. Sequential testing only works with the Z test. Not crossing the boundary means keep going, not that the variant does not work. This calculator has no futility boundary, the kind that would let you stop for lack of a chance to win.
When it misleads. The boundaries only protect you if the number of looks was set in advance and you only decide at those looks. Stopping at an unplanned look inflates the error again. A test that stops early by crossing a boundary tends to overstate the lift. The Bayesian method is not immune either. The probability still describes the data, but stopping on the first day it passes 95% raises the chance of calling a winner that is not better.
Results
Plan your sample before looking at the result.