Glossary
What is a confidence interval?
The range that tells you how big the effect might be, how it is calculated for two conversion rates, and the misreading almost everyone starts with.
Definition: A confidence interval is the range of values consistent with your data at a chosen confidence level, usually 95%. In an A/B test it is typically given for the uplift: an interval of +2% to +18% means the true improvement is probably somewhere in that range. An interval that includes zero means the test has not shown a real difference.
What it adds to a p-value
A p-value answers one yes-or-no question: is the gap distinguishable from zero? The interval answers the question you actually care about: how large is the effect, and how sure are we? Two tests can both be significant while one promises +1% to +3% and the other +1% to +40%. Only the interval shows that.
How it is calculated for two conversion rates
difference ± 1.96 × √( p₁(1 - p₁) / n₁ + p₂(1 - p₂) / n₂ )
p₁ and p₂ are the conversion rates, n₁ and n₂ the visitors. The square-root term is the standard error of the difference; 1.96 is the z-value for 95% (2.576 for 99%). Divide the ends of the interval by the control rate to express it as a relative uplift, which is how most tools show it.
Reading the width
A narrow interval comes from a large sample or a large effect; a wide one from few visitors or conversions. If the interval is too wide to make a decision, the answer is more data, not a different statistic. The width shrinks with the square root of the sample size: four times the visitors halves the interval.
The misreading to avoid
"There is a 95% probability that the true uplift is inside this interval" is the natural reading, and strictly it is the Bayesian one. The frequentist statement is about the procedure: intervals built this way contain the true value 95% of the time. In practice the two readings lead to the same decisions, which is why the shortcut survives; just do not treat the ends as hard limits.
A worked example
Control 3.0% of 4,000, variant 3.75% of 4,000. Standard error = √(0.03 × 0.97 / 4,000 + 0.0375 × 0.9625 / 4,000) = 0.00404. Interval = 0.0075 ± 1.96 × 0.00404, so -0.04 to +1.54 percentage points, or -1% to +51% relative to the 3.0% baseline. It includes zero: the variant may be much better, or no better at all. More visitors will narrow it.
Common mistakes
- Reporting the point estimate ("+25% uplift") without the interval around it.
- Treating an interval that barely excludes zero as a strong result.
- Reading the lower bound as a guaranteed minimum.
- Comparing intervals from tests run on different traffic and calling one change better than another.
In AB Test WP
AB Test WP Pro shows an uplift confidence interval for every running test alongside the significance result, so a wide interval is visible before anyone acts on the point estimate.
Related: A/B test significance calculator · A/B testing statistics, made simple · all glossary terms.
Common questions
What does 95% mean here?
That the method used to build the interval captures the true value in 95% of tests. It is a statement about the procedure. For decisions it is safe to read the interval as the plausible range of the effect.
Why does the interval include zero when the variant is ahead?
Because the lead is smaller than the noise at this sample size. The data are consistent with the variant being better and with the two being equal. Keep running to the planned sample.
Is a confidence interval the same as a margin of error?
The margin of error is half the width of the interval. "3.75% ± 0.6 points" and "3.15% to 4.35%" say the same thing.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.