Is your A/B test result real?
A difference is real when it is too big to be explained by chance at your sample size. With 1,000 visitors per version, 5.2% against 6.4% could easily be luck (p = 0.25). With 10,000 per version, the same rates are a clear win (p = 0.0003). Use the calculator below to check yours.
By Gevorg Petrosyan, founder of StatEVN · Updated 19 November 2026
Check your result
Enter the visitors and conversions for each version. Nothing you type leaves this page.
Same rates, different answer
Whether a difference is real depends on how many visitors are behind it. Here are the same conversion rates, 5.2% for A and 6.4% for B, with three sample sizes:
| Visitors per version | Conversions A / B | p-value | Verdict |
|---|---|---|---|
| 1,000 | 52 / 64 | 0.25 | Could easily be chance |
| 5,000 | 260 / 320 | 0.010 | Real at 95% confidence |
| 10,000 | 520 / 640 | 0.0003 | Real at 99% confidence |
B is 23% better in all three rows. Only the bigger tests can tell that apart from luck.
How the check works
The calculator runs a two-proportion z-test, the standard test for comparing two conversion rates. It asks: if A and B really converted at the same rate, how likely is a gap at least this big, just by chance? That probability is the p-value. Below 0.05, chance is an unlikely explanation and the difference is called significant at 95% confidence.
The range it shows is the 95% confidence interval for the difference. A range that stays above zero means B is better; a range that crosses zero means the test can't tell yet.
Five mistakes that make a result look real when it isn't
- Stopping when it looks good. Checking every day and stopping the first time p dips below 0.05 makes false wins far more likely. Decide the sample size first, then wait for it.
- Too few visitors. Small tests swing wildly, as the table above shows.
- Many versions or many metrics. Compare enough things and one will look significant by chance. With more than two versions, adjust for it.
- Uneven traffic. If you planned a 50/50 split but got 55/45, something in the setup may be favouring one version. Fix that before trusting the result.
- Too short a test. Run it for at least one full week, so weekdays and weekends are both in it.
How many visitors do you need?
A common rule of thumb for 95% confidence and 80% power: visitors per version ≈ 16 × p × (1 − p) ÷ d², where p is your current conversion rate and d is the smallest change you care about.
For a 5% conversion rate and a change of 1 percentage point (5% to 6%): 16 × 0.05 × 0.95 ÷ 0.01² ≈ 7,600 visitors per version. To spot half that change, you need four times as many.
Beyond the calculator
StatEVN's A/B test tool starts from the same totals and adds what a single number can't show: the range of the likely uplift, a plain-language verdict, and, if you upload one row per visitor, the check across segments such as device or country.
Questions
What confidence level should I use for an A/B test?
95% is the usual standard, meaning a p-value below 0.05. Use 99% when a wrong decision would be expensive or hard to undo.
Can I stop an A/B test as soon as it is significant?
No. Stopping the first time the result crosses the line greatly increases false wins. Decide the number of visitors before you start and analyse once you reach it.
What does the p-value mean in an A/B test?
It is the probability of seeing a difference at least as big as yours if the two versions actually performed the same. A small p-value means chance is an unlikely explanation.
Is a 20% uplift always worth it?
Only if it is real and the range around it is useful. With few visitors, a 20% uplift can sit inside a range that runs from a loss to a large gain.
Read next
Try it on a sample file
One click runs a real model on one of StatEVN's sample spreadsheets and explains it. No sign-up.