Statistical significance in A/B testing

Definition: A result is statistically significant when the difference between variants is large enough, given the sample size, that random variation alone would rarely produce it. It is usually declared at 95% confidence, meaning a p-value below 0.05. Significance says a difference is probably real; it does not say the difference is big or important.

What it measures

Start from the assumption that the two variants perform identically, the null hypothesis. Given that assumption and your sample size, the test works out how often chance alone would produce a gap at least as large as the one you observed. That frequency is the p-value. When it drops below a threshold you chose in advance, conventionally 5%, the result is called significant and the assumption of no difference is rejected.

Why 95%?

A 95% confidence level (a 5% significance level) is a convention, not a law of nature. It means you accept a one-in-twenty chance of calling a difference real when it is not. Tests that will trigger expensive changes sometimes use 99%; quick, cheap, reversible changes can justify 90%. Whatever you pick, pick it before the test starts.

How it is calculated for conversion rates

A/B testing tools compare two proportions with the two-proportion z-test: the gap between the rates is divided by the standard error of that gap, giving a z-score, and the z-score maps to a p-value. The significance calculator does this from four numbers: visitors and conversions for each variant.

What significance does not mean

• It is not the probability that the variant is better. A p-value of 0.04 does not mean a 96% chance B wins; that reading belongs to the Bayesian view, which computes a different quantity.
• It is not the size of the effect. With enough visitors a 0.1% improvement becomes significant; the confidence interval tells you whether it is worth acting on.
• It is not permanent. A significant result at day 10 that was not planned as a stopping point is often a false positive; see peeking below.

Why checking early breaks it

The 5% false-positive rate holds for one look at the data, at the planned sample size. Look every day and stop the first time the test crosses the line, and the false-positive rate climbs far above 5%, because you are giving chance many chances. Decide the sample size up front, run to it, read once.

A worked example

Control: 120 conversions from 4,000 visitors (3.0%). Variant: 150 from 4,000 (3.75%). Pooled rate 3.375%, standard error 0.00404, z = 1.86, two-sided p = 0.063. Not significant at 95%; the variant is ahead but the data are also consistent with no difference. Double the visitors with the same rates and z rises to 2.63, p = 0.009: significant. Sample size, not the size of the lead, made the difference.

Common mistakes

• Stopping at the first significant reading instead of the planned sample.
• Reading 95% confidence as a 95% chance that the variant is better.
• Treating a significant 0.5% lift as a business result without looking at the interval.
• Running a one-sided test after seeing which way the data lean.
• Ignoring a sample ratio mismatch, which invalidates the test whatever the p-value says.

In AB Test WP

AB Test WP Pro runs a two-sided two-proportion z-test and declares a winner only when both variants have at least 100 visitors, there are 25 conversions between them, the confidence is at or above your threshold (95% by default) and no sample-ratio-mismatch warning is active. Free shows the leader and the counts; nothing on Free can be read as significance.

Related: A/B test significance calculator · A/B testing statistics, made simple · all glossary terms.