Glossary
The two-proportion z-test
The test behind most A/B testing tools, worked through by hand once so the calculator's output makes sense.
Definition: The two-proportion z-test is the standard test for comparing two conversion rates. It measures how many standard errors apart the two rates are (the z-score) and converts that distance into a p-value. It assumes each visitor is an independent trial and works well once each variant has a few dozen conversions.
The formula
pooled p = (c₁ + c₂) ÷ (n₁ + n₂)
SE = √( p (1 - p) (1/n₁ + 1/n₂) )
z = (p₂ - p₁) ÷ SE
c is conversions, n is visitors, p₁ and p₂ the two rates. The pooled rate is used for the standard error because the test starts from the assumption that both variants share one true rate.
From z to p
The z-score is a distance in standard errors. Under the null hypothesis it follows the standard normal distribution, so the two-sided p-value is twice the tail area beyond |z|. Anchors worth remembering: z = 1.645 gives p = 0.10, z = 1.96 gives p = 0.05, z = 2.576 gives p = 0.01.
When it does not apply
- Very small counts: with fewer than about ten conversions in a variant the normal approximation is rough; wait for more data.
- Repeated looks: the p-value assumes one analysis at a planned sample. Daily checks inflate false positives.
- Non-independent visitors: the same person counted many times, or traffic surges from one source, break the independence assumption.
Chi-square or z-test?
For a 2 × 2 table (two variants, converted or not) the chi-square test and the two-proportion z-test give identical p-values; the chi-square statistic is z squared. Tools pick one or the other; the result is the same.
A worked example
Control: 120 of 4,000 (p₁ = 0.030). Variant: 150 of 4,000 (p₂ = 0.0375). Pooled p = 270 ÷ 8,000 = 0.03375. SE = √(0.03375 × 0.96625 × (1/4,000 + 1/4,000)) = 0.00404. z = 0.0075 ÷ 0.00404 = 1.86. Two-sided p = 0.063. Not significant at 95%. The calculator reproduces these figures from the four counts.
Common mistakes
- Using the unpooled standard error for the test (it belongs to the confidence interval).
- Reading a one-sided table for a two-sided test.
- Applying the test to tiny counts and trusting the third decimal.
- Recomputing daily and stopping at the first crossing.
In AB Test WP
AB Test WP Pro runs a two-sided two-proportion z-test on live results and shows the outcome as a confidence level, next to a Bayesian probability-to-beat and an uplift interval.
Related: A/B test significance calculator · A/B testing statistics, made simple · all glossary terms.
Common questions
Why is the rate pooled?
Because the null hypothesis says both variants share one true conversion rate, and the best estimate of that shared rate uses all the data. The confidence interval for the difference uses the unpooled rates instead.
What z-score is significant?
1.96 or more (in either direction) for 95% two-sided confidence; 2.576 for 99%.
Is a chi-square test different?
For two variants and a yes-or-no outcome it is the same test; the chi-square statistic equals z squared and the p-values match.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.