Frequentist vs Bayesian A/B testing

Definition: Frequentist A/B testing asks how unlikely your data would be if the variants were equal and reports a p-value and confidence. Bayesian A/B testing asks how probable it is that the variant beats the control given the data and reports a probability, such as a 96% chance to beat. Both use the same counts; they answer different questions and, with sensible priors and full samples, usually agree.

The frequentist reading

Start from "no difference" and ask how surprising the data are. The answer is the p-value; below 0.05 the result is called significant. The strength of this approach is a guaranteed false-positive rate, provided you fix the sample size in advance and read the result once. Its weakness is that the number it produces is not the one people want, which is the probability that B is better.

The Bayesian reading

Start from a prior belief about the conversion rates, update it with the observed counts, and read off the probability that B's rate exceeds A's, plus the distribution of the uplift. The output answers the natural question directly: "92% chance B is better, expected lift 8%." Its weakness is that the prior is a choice, and the numbers can be read too casually when the sample is small.

Where they differ in practice

• Interpretation. 95% confidence is not a 95% chance of winning; 95% probability to beat is exactly that, given the model.
• Peeking. Bayesian tests are often sold as safe to read continuously. They are more forgiving, not immune: stopping the moment the probability crosses a line still favours lucky streaks.
• Small samples. A probability-to-beat of 80% after 60 visitors is a weak signal that looks like a strong one. Frequentist thresholds refuse to speak until the sample is in.
• Priors. An informative prior ("changes like this rarely move the rate by more than 10%") makes Bayesian results more realistic; a flat prior makes them optimistic.

Which to use

Report both if the tool offers both, decide on a rule written before the test starts, and let the planned sample size govern when you look. Where the two disagree, the sample is almost always too small.

A worked example

Control 120 of 4,000, variant 150 of 4,000. Frequentist: z = 1.86, two-sided p = 0.063, not significant at 95%. Bayesian with a flat prior: roughly a 97% probability that the variant's true rate is higher, with an expected uplift near 25% and a wide credible interval. Same data. The frequentist number says "could be chance"; the Bayesian number says "probably better, but the interval is wide". A team that stops here on the Bayesian reading is taking a real risk it has not measured.

Common mistakes

• Reading 95% confidence as a 95% chance of winning.
• Treating a Bayesian probability-to-beat as immune to early stopping.
• Switching schools mid-test to whichever reports the better number.
• Using a flat prior on a small sample and believing the result.

In AB Test WP

AB Test WP Pro shows the two-sided significance result, a Bayesian probability-to-beat and an uplift confidence interval together, and declares a winner only when both variants have at least 100 visitors, there are 25 conversions between them and the confidence threshold (95% by default) is met.

Related: A/B test significance calculator · A/B testing statistics, made simple · all glossary terms.