A/B Test Significance Calculator

The Complete A/B Test Significance Guide

How an A/B test decides whether one version really beats another: p-values, confidence levels, one-tailed and two-tailed tests, confidence intervals, and how many visitors you need before you start.

Open A/B Test Calculator

Overview: What an A/B Test Result Really Says

An A/B test splits visitors at random between a control (A) and a challenger (B), then compares how many people in each group completed the goal. The raw difference in rates is not enough to act on. Two identical versions will still show different rates by chance, so the question is whether the gap you measured is bigger than luck would normally produce.

The calculator answers that question with a pooled two-proportion z-test. It assumes both versions share one underlying rate, measures how far your observed gap sits from zero in standard-error units, and reports the result as a p-value. It also reports a confidence interval for the difference, which shows the range of gaps consistent with your data. Both numbers matter: the p-value says whether to believe there is a difference, and the interval says how big it plausibly is.

Control Walkthrough

The tool has two tabs. The Test results tab checks an experiment that has already run. The Sample size planner tab tells you how many visitors you need before it starts. Work through the results tab first.

ControlWhat it does
Visitors (A, B)Whole number above zero for each variant. The “Conversion rate” line under the variant shows the rate as you type.
Conversions (A, B)Whole number, zero or more, and never more than the visitor count. Anything else turns the field red and holds back the verdict.
Add a third variant (C)Reveals a C column. If C is empty, it is pre-filled with a sample so you can see the three-way layout. C is always compared against A on its own.
Remove (cross icon on C)Hides variant C again and drops it from the verdict and the chart.
Confidence level90%, 95% or 99%. Higher levels demand a smaller p-value before a result counts as significant, and widen the intervals.
Test directionTwo-tailed (default) or one-tailed. This changes the p-value and the interval width used in the verdict.
Load sample / Clear figuresLoad sample fills A and B with 10,000 visitors each and 320 and 372 conversions, and sets 95% and two-tailed. Clear figures empties the counts but keeps your settings.
Verdict panelShows the chip (significant or not yet significant), the relative uplift of B over A, a plain-language sentence, and the confidence reached.
Chart and breakdown tableBars show each conversion rate with whiskers for its interval. The table repeats visitors, conversions, rate and interval for each variant.
Copy summaryCopies the verdict, the rates, the z-score and the p-value as plain text.

On the planner tab, enter your current conversion rate as a percentage and the smallest relative lift worth detecting. Visitors per day is optional. Significance level, statistical power, number of variants and test direction set the statistical strictness. The result shows visitors per variant, total visitors, and days at your traffic level, rounded up to whole weeks. The table below lists other lift sizes, and Copy plan saves the summary.

Worked Examples

Example 1 - a clear win (the default sample). Variant A converted 320 of 10,000 visitors, a rate of 3.20%. Variant B converted 372 of 10,000, a rate of 3.72%. The absolute uplift is +0.52 percentage points, a relative uplift of roughly +16%. The pooled z-score is 2.012, giving a two-tailed p-value of 0.0442. The confidence reached is 95.6%, which clears the 95% threshold, so the verdict is significant. The 95% interval for the difference runs from +0.01 to +1.03 percentage points and does not include zero. Switching to a one-tailed test halves the p-value to 0.0221, which is also significant.

Example 2 - a promising gap that is not yet proven. Variant A converted 30 of 1,000 visitors (3.00%) and variant B converted 36 of 1,000 (3.60%). The relative uplift is +20%, but the two-tailed p-value is 0.4526 and the confidence reached is only 54.7%. At 95% the verdict reads not yet significant. The interval for the difference runs from -0.97 to +2.17 percentage points, which includes zero, so the data cannot rule out no difference. The honest action is to keep the test running to its planned size.

Example 3 - planning the test. Using the planner defaults, a 3.2% baseline and a 10% relative lift give a target rate of 3.52%. At 95% significance, 80% power and two-tailed testing, each variant needs 49,777 visitors, or 99,554 across both. At 3,000 visitors a day, that is 34 days, which rounds up to 5 full weeks. The reference table shows that a 5% lift would need 194,530 visitors per variant (130 days), while a 20% lift needs 13,015 (9 days). Small lifts are expensive to detect, which is why the lift you pick matters so much.

Tips and Common Mistakes

Plan firstRun the planner before the test starts and write the target visitor count down. This is the single most useful habit for trustworthy results.
Do not peek and stopChecking every day and stopping as soon as the result turns significant inflates false winners. Wait for the planned sample.
Count the same wayUse the same unit for both variants, such as unique users, and the same time window. Mixed counting makes the rates incomparable.
Read the interval tooOverlapping whiskers do not prove no difference. The interval for the difference is the number that tells you whether zero is plausible.

Watch out for small counts. When a variant has fewer than about 5 conversions or non-conversions, the normal approximation behind the test becomes unreliable, and the calculator shows a warning. Adding a third variant also raises the chance that one comparison looks significant by luck, so treat a lone C win with caution. Finally, statistical significance is not the same as business value: a real but tiny lift may not justify the cost of the change.

Frequently Asked Questions

What does the p-value actually mean?

It is the probability of seeing a difference at least as large as the one you measured if the two versions truly converted at the same rate. A small p-value means the observed gap would be unusual under that assumption. It is not the probability that variant B is better, and it does not measure how large the effect is.

Why does the planner ask for a relative lift instead of a percentage-point change?

A relative lift is measured against your baseline, so a 10% lift on a 3.2% rate means a target of 3.52%, a change of 0.32 percentage points. Relative lifts stay sensible across products with different baseline rates, and the planner shows the resulting target rate in its table.

Can I use the calculator to check a test that has already finished?

Yes. The results tab works from whatever counts you enter. The p-value is only meaningful if the test ran to a fixed plan, so make sure you did not stop it early because the result looked good.

Why does the p-value change when I switch between one-tailed and two-tailed?

A one-tailed p-value is exactly half the two-tailed value for the same data, because the two-tailed version counts differences in both directions. In the 3.20% versus 3.72% example, the p-value moves from 0.0442 to 0.0221. Pick the direction before you run the test, not after you see the data.

Open A/B Test Calculator