A/B Test Significance Calculator
Free A/B Test Significance Calculator, know whether your winner is real or just noise
Enter the visitors and conversions for your control and your variant and get a proper statistical read: each version's conversion rate, the absolute and relative lift, the p-value from a real two-proportion z-test, and a confidence interval on the difference. Instead of leaving you to interpret a p-value, the tool says in plain words whether the result is significant at the level you choose, and warns you when the sample is simply too small to trust. Everything runs in your browser.
- 100% free
- No sign-up
- Nothing uploaded
- Runs in your browser
A/B testing is supposed to replace opinion with evidence, and most of the time it quietly replaces it with a worse opinion dressed up as evidence. The variant is a couple of points ahead after a few hundred visitors, someone calls it a win, the change ships, and the lift evaporates the following month because it was never real to begin with. The trap is that small differences look meaningful when you want them to, and a raw conversion rate cannot tell you whether a gap is a genuine effect or the ordinary wobble you would see from flipping the same coin twice. The tool that answers that question is a significance test, but significance tests come wrapped in jargon, null hypotheses, z-scores, two-tailed p-values, that sends people back to declaring winners by eye. Worse, the one number most people do take away, the p-value, is the single most misunderstood figure in marketing: a significant p-value on a tiny sample is not a result, it is a coincidence with good marketing. This tool does the real statistics and then translates them. Enter four numbers and it runs a proper two-proportion z-test, tells you whether the difference clears the confidence level you set, and, crucially, tells you when the sample is too small for any of it to mean anything. The page is free and public; you sign in only to copy the result out.
What it does, and why it's built this way
What statistical significance actually tells you
When two versions of a page produce two different conversion rates, there are two possible reasons: either one version genuinely converts better, or they are really the same and you are looking at random variation. A significance test measures how easily the second explanation accounts for what you saw. The p-value it produces is the probability that a difference at least this large would appear by pure chance if the two versions were actually identical. A small p-value means chance is a poor explanation, so the difference is probably real; a large one means chance explains it comfortably, so you have not learned anything yet. The confidence level, 90, 95 or 99 per cent, is simply the bar you set for how much doubt you are willing to accept before calling a winner. This tool computes the p-value with a genuine two-proportion z-test and compares it to your chosen level, so the verdict is a real statistical statement rather than a hunch. It also shows a confidence interval on the difference, which is often more useful than the p-value: if that range crosses zero, the true difference could be nothing.
Why sample size decides everything
The most dangerous test is a small one that looks significant, and it is dangerous precisely because the maths cooperates. With very few visitors, a run of luck on either side can push the p-value below your threshold even though nothing real is happening, which is how teams end up shipping changes that do nothing and then quietly hurt them. A significance test assumes you have gathered enough data for the normal approximation behind it to hold, and below roughly a hundred visitors or a handful of conversions per side, that assumption breaks and the p-value stops being trustworthy. This is the part other calculators leave out, and it is the part that causes the most damage. So alongside the statistics, this tool checks the size of your sample and warns you outright when it is too small, no matter how confident the p-value looks. A real read needs both a low p-value and enough data to believe it, and the tool insists on both rather than letting a lucky small test masquerade as a discovery.
Real statistics, in your browser, kept private
The two-proportion z-test, the normal distribution behind the p-value and the confidence interval are all computed on your own machine, in the browser, with no data sent anywhere. The visitor and conversion counts you enter are used to do the maths locally and are never logged or stored against your account, which is as it should be, because those numbers describe your business. Results update the instant you change an input, so you can see how many more visitors it would take to reach significance by nudging the numbers and watching the verdict change. The whole page, the explanation and the calculator are free and public, because a tool that stops people shipping fake wins is exactly the kind of resource the platform is meant to offer freely. The only thing that asks for a free account is copying the result out to share, which keeps the public endpoint from being scripted against, and it is the same soft sign-in used across the site.
How to use it
- 1
Enter the control
Fill in the control's visitors, everyone who saw the original version, and its conversions, how many of them took the action you are testing. The tool shows the control's conversion rate straight away so you can sanity-check the inputs.
- 2
Enter the variant
Do the same for the variant: the visitors who saw the new version and the conversions it produced. As soon as both sides have numbers, the calculator runs the test and fills in the lift, p-value and confidence interval.
- 3
Choose your confidence level
Pick the bar the result must clear: 95 per cent is the common default, 90 is more lenient and 99 more strict. This sets how little room for chance you are willing to accept before you are prepared to call a winner.
- 4
Read the verdict, not just the p-value
Read the plain-language verdict: it states whether the result is significant, which version is ahead, and whether the sample is large enough to trust. If it flags the sample as too small, keep the test running before you act. Sign in once to copy the full result.
Common mistakes
Calling a winner too early
Ending a test the moment the variant edges ahead is how noise gets shipped as a result. Decide on a sample size or a run length before you start and hold to it. Let the verdict here confirm both that the difference is significant and that the sample is big enough before you declare anything.
Trusting a significant p-value on a tiny sample
A low p-value from a few dozen visitors is very often luck, not signal. Significance tests need enough data for the maths behind them to hold. This tool warns you when the sample is too small precisely because that warning is the one most likely to save you from a false win; heed it even when the p-value looks convincing.
Peeking and stopping the instant it turns significant
Checking a running test repeatedly and stopping the first time it crosses your threshold inflates your real error rate well beyond the level you chose, because with enough peeks a true tie will cross the line eventually by chance. Check at your planned checkpoints, not continuously, and treat an early crossing with suspicion rather than relief.
Reading relative uplift without the absolute numbers
A headline like a fifty per cent uplift can come from a move of two conversions out of four, which means almost nothing. Always read the relative lift next to the absolute difference and the raw counts. This tool shows all three together so a dramatic-sounding percentage built on a handful of events cannot mislead you on its own.
Frequently asked questions
- What does statistical significance mean here?
- It means the difference between your two versions is large enough, given how many people saw each, that it is unlikely to be down to chance. The tool runs a two-proportion z-test and produces a p-value, the probability of seeing a gap this big if the versions were truly equal. When that probability falls below the level you set, the result is called significant. It is a statement about how well random variation explains your data, not a guarantee the effect will hold forever.
- What confidence level should I use?
- Ninety-five per cent is the standard default and a sensible choice for most marketing tests. Use ninety per cent if the cost of a wrong call is low and you want to move faster, and ninety-nine per cent when the decision is expensive or hard to reverse and you want to be more certain before acting. The level is simply how much risk of a false positive you are willing to accept, so match it to the stakes of the change you are testing.
- How large a sample do I need?
- It depends on how big the real difference is: small effects need much larger samples to detect. As a floor, treat anything under about a hundred visitors or a handful of conversions per version as too small to trust, which is exactly when this tool starts warning you. Rather than a fixed number, watch the verdict: if it still flags the sample as small, you need more data, and if the result is not yet significant, running longer is the honest next step.
- What is the difference between absolute and relative uplift?
- Absolute uplift is the gap in percentage points, for example moving from 4 per cent to 5 per cent is one point. Relative uplift expresses that same gap as a proportion of the starting rate, so one point on top of 4 per cent is a twenty-five per cent relative increase. Relative figures sound more impressive and are easy to inflate on small numbers, so this tool shows both alongside the raw counts to keep the picture honest.
- Does this work for tests with more than two variants?
- This calculator compares two versions at a time, a control and a single variant, which is the classic A/B test. For a test with several variants at once, comparing each against the control separately raises the chance that one looks significant by luck, so you would need to adjust your threshold for the number of comparisons. For most purposes, running each variant against the control here and treating borderline results with extra caution is a reasonable approach.
- Is the calculator free and private?
- Yes on both counts. The tool and all of its guidance are free, and the statistics run entirely in your browser, so your visitor and conversion numbers are never sent to a server or stored against your account. A free account is required only to copy the result out to share, which keeps the public tool from being scripted against at scale. Using the calculator on screen needs no account at all.
Need this done properly, at scale?
The tool handles the one-off. When it's a system you're building, that's the paid version of the job, and we do that too.
SEO Services