Test analysis
Compare versions. Understand the evidence. Choose your next step.
1. Test duration
2. Test data
| Group | Users | Conversions | Intended split % | Conversion rate | Relative lift | P(beat control) | Remove |
|---|---|---|---|---|---|---|---|
% | Conversion rate— | Relative lift— | P(beat control)— | ||||
% | Conversion rate— | Relative lift— | P(beat control)— |
Confidence and power
A fire alarm can fail in two ways. It can ring with no fire, or stay silent during one. Confidence controls the first mistake and power controls the second. Making the alarm stricter cuts false alarms but raises the risk of missing a small fire. To reduce both at once, the test needs more data. In your test, confidence is 95% and power is 80%. If the variant has the target lift, the chance that the test detects it is 80%, as long as it runs to the planned sample.
see details
A type I error is calling a difference that does not exist, a false positive. Its probability is alpha, α = 1 − confidence. With Bonferroni on and 3 or more groups, each comparison uses α' = (1 − confidence) / (groups − 1).
A type II error is missing a difference that does exist, a false negative. Its probability is β = 1 − power. Power always refers to a specific effect size. 80% power means an 80% chance of detecting the difference if the true lift equals the target lift. For smaller true lifts, power is lower.
Both errors compete for the same sample, which comes from n = (zα × √(2 × p̄ × (1 − p̄)) + zβ × √(p1 × (1 − p1) + p2 × (1 − p2)))² / (p2 − p1)². Here p1 is the baseline rate, p2 = p1 × (1 + lift), p̄ = (p1 + p2) / 2, zβ = z(power), and zα = z(1 − α'/2) two-sided or z(1 − α') one-sided. With a fixed sample, raising confidence lowers power. To keep both, the sample has to grow. At 80% power, going from 95% to 99% confidence takes about 50% more users. At 95% confidence, going from 80% to 90% power takes about 34% more.
In the Bayesian method, confidence is the P(beat) threshold for calling a winner. There is no power there, but the "underpowered" verdict uses the same sample rule as the Z test.
When it misleads. 95% confidence does not mean a 95% chance that the result is right. If few of the ideas you test really work, far more than 5% of declared winners can be false. Power is not the chance that the result is right either. It is a property of the plan, computed before the test. Observed power, Φ(|z| − critical z), is just the p-value written another way. It sits near 50% when the p-value is right at the threshold and says nothing about whether the sample was large enough. For that, compare the achieved MDE with the target lift.
Peeking and sequential testing
Imagine you could blow the final whistle the moment your team takes the lead. You would win games you would lose over the full 90 minutes. Checking a test every day and stopping at the first significant result works the same way. Every look is another chance for luck to pass as a real effect. A sequential test agrees in advance on how many times you will look and demands a much bigger lead to stop early.
see details
Why peeking inflates false positives. When there is no effect, each analysis has an alpha-sized chance of coming out significant by luck. Looking several times and stopping at the first significant result piles those chances up. At 95% two-sided with the usual threshold at every look, the overall false positive rate climbs to about 8% with 2 looks, 14% with 5 and 19% with 10. With endless looks, it heads toward 100%.
How the boundaries fix it. A sequential test spends alpha gradually, using the Lan-DeMets O'Brien-Fleming-type spending function. One-sided tests use α(t) = 2 − 2 × Φ(z(1 − α/2) / √t). Two-sided tests spend half of alpha on each side with the same function, α_side(t) = 2 − 2 × Φ(z(1 − α/4) / √t), the gsDesign and ldbounds convention. Here t is the information fraction, current n / maximum planned n. The boundaries come from that function by recursive numerical integration, the Armitage, McPherson and Rowe method. By the last look, the alpha spent adds up to exactly the alpha you chose. One-sided tests use the same idea with a single upper boundary. With Bonferroni on, the correction comes first and the corrected alpha is then spread across the looks.
The early boundaries are very strict, and the last one sits close to the usual critical z. With 4 equally spaced looks at 95% two-sided, the boundaries are about z = 4.33, 2.96, 2.36 and 2.01. In exchange, the maximum sample grows only a little. With 5 looks and 80% power, it is about 2.5% larger than the fixed-sample test.
The calculator uses the actual sample at each look to compute t, not the planned one. That keeps alpha correct if a look comes early or late. Sequential testing only works with the Z test. Not crossing the boundary means keep going, not that the variant does not work. This calculator has no futility boundary, the kind that would let you stop for lack of a chance to win.
When it misleads. The boundaries only protect you if the number of looks was set in advance and you only decide at those looks. Stopping at an unplanned look inflates the error again. A test that stops early by crossing a boundary tends to overstate the lift. The Bayesian method is not immune either. The probability still describes the data, but stopping on the first day it passes 95% raises the chance of calling a winner that is not better.
Results
Enter group data to see the result.
Alerts
—
TL;DR
Request an interpretation of the numbers and alerts. The calculation is already complete in your browser.
Bayesian method
Think of a weather forecast. It does not promise sunshine. It says there is a 20% chance of rain. The Bayesian method does the same for your test. It starts with no preference for any conversion rate and updates that belief with every user who arrives. In the end it answers the question you actually have. A 97% chance of being better means that, given the data, the variant converts more than the control in 97 out of 100 plausible scenarios.
see details
Each group starts with a Beta(1, 1) prior, which gives equal weight to every conversion rate between 0% and 100%. The data update that belief into the posterior Beta(1 + conversions, 1 + users − conversions). The prior is worth one extra conversion and one extra non-conversion, so it carries little weight once each group has a few dozen conversions.
The calculator draws 50,000 values from each group's posterior, with a seed derived from the test's numbers. The same numbers always give the same result. The probability to beat the control is the share of draws where the variant beats it, P(beat) = P(CRv > CRc). With 3 or more groups, the probability to be best is the share of draws where a group has the highest rate of all.
The expected loss is E[max(CRc − CRv, 0)]. It is the average conversion you lose by shipping the variant, counting the scenarios where it is better as zero. It is shown in percentage points and relative to the control's rate.
The credible interval of the lift runs from the (1 − confidence) / 2 quantile to the 1 − (1 − confidence) / 2 quantile of (CRv − CRc) / CRc. Under the model, the true lift lies inside it with probability equal to the confidence level. That reading holds for the Bayesian interval. It does not hold for the Z test's confidence interval.
A variant wins when P(beat) ≥ confidence and loses when P(beat) ≤ 1 − confidence. There is no p-value and no significance test. The answer is already a probability about the business question, and the 95% threshold is the decision rule you picked. That is why the Bayesian method ignores the one-sided or two-sided choice and the Bonferroni correction.
When it misleads. The probability is only as good as the model. The model assumes each user converts independently, at a rate that stays fixed during the test, and that group assignment worked. Sample ratio mismatch, novelty effects and seasonality break those assumptions. Checking every day and stopping when P(beat) passes 95% also raises the chance of calling a winner that is not better. The posterior still describes the data, but that stopping rule loses its error control, and the flat prior does not protect you. With small samples, P(beat) swings a lot from one day to the next, so check the interval and the expected loss too. With several variants, the chance that one crosses the threshold by luck grows, even though Bonferroni is not part of the math.
Z test
Picture a coin that might be loaded. You start by assuming it is fair and flip it 100 times. If you get 70 heads, a fair coin would almost never do that, and your suspicion grows. The Z test does the same with your test. It assumes the variant and the control convert at the same rate and measures how unusual a difference at least as large as the observed one would be. That measure is the p-value. The smaller the p-value, the harder it is to explain the difference by chance alone.
see details
The null hypothesis H0 says the variant's true rate equals the control's. The test measures how far the data sit from that hypothesis, in standard errors.
For each group, CR = conversions / users and SE = √(CR × (1 − CR) / users). The standard error of the difference is SEdiff = √(SEc² + SEv²), unpooled, as in ABTestGuide. The Z-score is z = (CRv − CRc) / SEdiff.
The two-sided p-value is 2 × (1 − Φ(|z|)) and the one-sided p-value is 1 − Φ(z), where Φ is the cumulative normal distribution. The result is significant when p-value < α'. Alpha is α' = 1 − confidence, divided by groups − 1 when the Bonferroni correction is on.
The confidence interval of the difference is (CRv − CRc) ± z(1 − α'/2) × SEdiff. For a one-sided test, the calculator shows only the lower bound. The relative lift interval divides those bounds by CRc, an approximation that ignores the uncertainty in the control's rate. A 95% interval does not mean a 95% chance that the true value is inside it. It means that if you repeated the test many times, 95% of the intervals built this way would contain the true value.
The test relies on the normal approximation, which needs a large sample. A rule of thumb is at least 10 conversions and 10 non-conversions in each group. At 0% or 100% conversion, a group's standard error becomes zero and the test understates the uncertainty.
When it misleads. The p-value only holds if the sample size was fixed in advance and the test is analyzed once. Peeking and stopping at significance inflates false positives. To monitor a running test, use sequential mode. Testing several variants or several metrics also multiplies the chances of a false positive. A significant result does not mean the effect is large. With enough traffic, differences too small to matter become significant. A non-significant result does not prove the rates are equal either. It only shows the data are not enough to rule out equality.
Bayesian or Z test?
Both methods look at the same data and ask different questions. The Z test works like a court. It presumes the variant changes nothing and drops that presumption only with strong evidence. The Bayesian method works like someone pricing a bet. It tells you the chance that the variant is better. Far from the decision threshold, the two almost always agree. Close to it, one can say "winner" while the other says "inconclusive".
see details
Use the Bayesian method when the question is which version to ship. It gives the probability that the variant is better and the expected loss if you are wrong, two numbers that map well onto a business decision. It is the calculator's default.
Use the Z test when you need explicit control of the false positive rate, when you planned the sample with confidence and power, or when you will monitor the test while it runs. Sequential testing with O'Brien-Fleming boundaries is only available for the Z test. It also reproduces the numbers of tools like ABTestGuide.
Why they disagree. With a Beta(1, 1) prior and large samples, P(beat) comes close to 1 − one-sided p-value, which is Φ(z). At 95% confidence, the Bayesian method calls a winner when Φ(z) ≥ 0.95, that is, when z ≥ 1.645. The two-sided Z test needs |z| ≥ 1.96. Between those values, the Bayesian method sees a winner and the Z test sees no significant difference. For example, z = 1.75 gives a P(beat) near 96% and a two-sided p-value near 0.08.
The Bonferroni correction widens that gap. With 3 or more groups, the Z test gets stricter while the Bayesian threshold stays put. A disagreement is not a calculation error. It shows that the evidence is borderline and that the decision depends on the rule you chose.
3. Details and calculators
Revenue impact
Enter test duration and value per conversion.