A/B test significance calculator with Bayesian probability
This A/B test significance calculator tells you whether the difference between a control (A) and a variant (B) is bigger than chance would explain. Enter visitors and conversions for each version and it runs a two-proportion z-test, then adds the figures people actually ask for: the relative uplift with a range, the difference in percentage points, and the Bayesian probability that B beats A.
The frequentist and Bayesian answers sit side by side so you can see whether they agree. Everything is computed in your browser from the numbers you type, and the formulas are listed below the results.
How to use the A/B Test Significance Calculator
- Enter visitors and conversions for Control (A) and Variant (B), counted over the same period.
- Pick a confidence level (95% is the usual choice) and whether you want a two-sided or one-sided test.
- Leave the Bayesian prior at 1 and 1 unless you have a reason to bring in earlier knowledge.
- Read the verdict, the p-value, the uplift range and the probability that B beats A.
- Check any warning about small counts before relying on the answer.
A/B Test Significance Calculator features
Two-proportion z-test
Pooled-variance test for equal conversion rates, two-sided or one-sided.
Uplift and difference with ranges
Relative uplift by the log method and the absolute difference with a Wald interval.
Bayesian probability to beat control
Beta-Binomial posteriors with an editable prior, integrated numerically.
Credible intervals
The plausible range of each version’s true conversion rate.
Worked example: 12.0% against 15.0%
Control converts 120 of 1,000 visitors (12.0%) and the variant 150 of 1,000 (15.0%). The pooled rate is 270 ÷ 2,000 = 13.5%, the standard error is √(0.135 × 0.865 × (1/1,000 + 1/1,000)) = 0.0153, so z = 0.03 ÷ 0.0153 = 1.96 and the two-sided p-value is about 0.0496. This is the textbook example given for the two-proportion z-test; the result is only just significant at 95%, and the range for the difference, 0.01 to 5.99 points, nearly touches zero.
The Bayesian answer is a different kind of statement. With a flat prior, the probability that B truly converts better than A is about 97.5%. It is not a p-value and it does not say how much better B is; for that, read the uplift and credible interval.
Which result to trust
Peeking is the main trap. If you check daily and stop at the first significant result, the real false-positive rate is much higher than 5%. Fix the sample size beforehand with the sample size calculator, and test many variants or metrics only with a stricter threshold.
| Output | Answers | Watch out for |
|---|---|---|
| p-value | How surprising this gap would be if A and B were identical | It is not the chance that B is better |
| Uplift range | How large the real improvement might be | A range that includes 0% means no clear winner |
| Chance B beats A | Probability B’s true rate is higher, given the data and the prior | Depends on the prior; strong priors need justification |
Limits
- The z-test needs enough conversions: when expected counts fall below 5 a warning appears and an exact test is the better choice.
- It assumes visitors are randomly split and counted once. Repeat visitors, bots and uneven traffic mixes can bias the result.
- Statistical significance does not mean business significance: check the size of the uplift and its range.
- A test for a click-through rate can use the same numbers; see the CTR calculator.
A/B Test Significance Calculator FAQ
How do I know if my A/B test is statistically significant?
Compare the p-value with 1 minus your confidence level: below 0.05 for 95% confidence. The verdict above does this for you, but also look at the uplift range and the sample size.
What is the Bayesian probability to beat control?
It is the probability, given your data and prior, that the variant’s true conversion rate is higher than the control’s. It is computed from Beta posteriors for each version.
What prior should I use?
Beta(1, 1), the default, treats every conversion rate as equally likely and lets the data decide. Use another prior only if you have solid earlier evidence.
Two-sided or one-sided?
Use two-sided unless you would act on a loss in exactly the same way as on no change. One-sided gives a smaller p-value for the same data, which makes it easy to misuse.
Why does it say no clear winner?
The observed gap could plausibly come from random variation at your sample size. Collect more data up to the planned sample size, then decide.
Can I use this for email or ad clicks?
Yes. Treat clicks as conversions and delivered emails or impressions as visitors.