Glossary
What is a p-value?
The definition that statisticians accept, the readings that they do not, and how the number gets from your conversion counts to a confidence level.
Definition: A p-value is the probability of seeing a difference at least as large as the one in your test if the two variants truly performed the same. A small p-value, conventionally below 0.05, means the observed gap would be unusual under that assumption, so the difference is called statistically significant. It is not the probability that the variant wins.
The precise definition
Assume the null hypothesis: the variants convert at the same rate and the gap you see is noise. Under that assumption, and given your sample sizes, the sampling distribution of the gap is known. The p-value is the share of that distribution at least as far from zero as your observed gap. A p-value of 0.03 means that, if there were truly no difference, only 3 in 100 tests would show a gap this large or larger.
Three things a p-value is not
- Not the probability that there is no difference. It is a probability about the data given an assumption, not about the assumption given the data.
- Not the probability that B beats A. That quantity exists, but it comes from a Bayesian calculation, and it is a different number.
- Not a measure of effect size. A tiny effect with a huge sample can have a very small p-value.
One-sided or two-sided
A two-sided test asks whether the variants differ in either direction; a one-sided test asks only whether B is better. One-sided p-values are half as large for the same data, which is why they are tempting and why choosing one after seeing the results is a classic way to fool yourself. Most A/B testing tools, and AB Test WP, report two-sided values.
From p-value to confidence
Tools often show "confidence" as 1 minus the p-value: p = 0.03 appears as 97% confidence. That is a convenient display, not a probability that the variant is better. The confidence interval is the more useful companion, because it shows the range of plausible effect sizes.
two-sided p from z: z = 1.645 → p = 0.10 · z = 1.96 → p = 0.05 · z = 2.576 → p = 0.01
A worked example
Control 120 of 4,000 (3.0%), variant 150 of 4,000 (3.75%). The z-test gives z = 1.86. In the standard normal distribution, 3.15% of values lie above 1.86 and 3.15% below -1.86, so the two-sided p-value is 0.063. Not below 0.05, so not significant at 95%. The calculator returns the same figures.
Common mistakes
- Treating p = 0.049 as a win and p = 0.051 as a loss. They are the same evidence; the line is a convention.
- Reporting a one-sided p-value because the two-sided one missed the threshold.
- Computing a fresh p-value every day and stopping at the first small one.
- Reading a small p-value as a large improvement.
In AB Test WP
AB Test WP Pro reports the two-sided result as a confidence level next to a Bayesian probability-to-beat and an uplift interval, so the three readings of a result sit side by side rather than being confused for one another.
Related: A/B test significance calculator · A/B testing statistics, made simple · all glossary terms.
Common questions
Is 0.05 a magic line?
No. It is the conventional threshold for 95% confidence. The evidence at p = 0.04 and p = 0.06 is almost identical; what matters is that you set the threshold before the test and honour it.
Does a lower p-value mean a bigger effect?
No. It means the observed gap is less likely under the assumption of no difference, which depends on sample size as much as on the gap. Use the confidence interval for the size of the effect.
What is the difference between p-value and confidence?
Confidence, as tools display it, is one minus the p-value. A confidence level is the threshold you chose (95%); a confidence interval is the range of effect sizes consistent with the data.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.