Glossary
What is A/B testing?
The definition, how a test actually works, a worked example with real numbers, and the mistakes that turn a good idea into a false result.
Definition: A/B testing shows two versions of a page or element to different visitors at the same time and measures which one produces more of a chosen action, such as sign-ups or purchases. Visitors are assigned at random, so the only systematic difference between the two groups is the change being tested.
How an A/B test works
Every A/B test has the same four parts. A control, the version you have today. A variant, the version with one deliberate change. A goal, the action you count, such as a purchase or a form submission. And random assignment: each visitor is put into one group by chance and stays there on later visits.
The test runs until each group has enough visitors for the comparison to mean something, then the conversion rates are compared. If the gap is larger than random variation would produce, the variant is declared the winner and becomes the new control.
Why random assignment is the whole point
Without randomisation you are comparing different audiences, not different designs. Traffic on Monday differs from traffic on Saturday; visitors from an email campaign differ from visitors from search. Splitting each visitor at random, at the same moment, makes the two groups statistically the same in every respect except the change, so any difference in results can be credited to the change.
This is what separates an A/B test from a before-and-after comparison, where a redesign launched in November looks brilliant because November traffic buys more.
What you can test
Anything a visitor sees and that has a measurable outcome: a headline, a call-to-action button, the order of sections, a pricing table, a product image, an entire page design. Tests of one element within a page are called element tests; tests that send visitors to two separate pages are URL split tests. Tests with more than two versions are A/B/n tests.
A worked example
A pricing page gets 8,000 visitors in two weeks, split evenly. The control (A) turns 120 of its 4,000 visitors into trials, a conversion rate of 3.0%. The variant (B), with a comparison table added, converts 150 of 4,000, or 3.75%. That is a relative uplift of 25%, and it looks like a clear win.
Run the numbers through a two-proportion z-test and the picture changes: z = 1.86, p = 0.063. At the usual 95% bar that is not yet statistically significant. The confidence interval for the difference runs from -0.04 to +1.54 percentage points; it includes zero. With the same rates and twice the visitors it would be significant. The lesson is that a lead is not a result until the planned sample is in.
Common mistakes
- Stopping the moment one version is ahead. Early numbers swing wildly; decide the sample size first and wait for it.
- Changing several things at once, then not knowing which one mattered.
- Running a test for three days. Weekday and weekend visitors behave differently; run whole weeks.
- Testing tiny changes on a low-traffic page. Small effects need enormous samples; test bold changes instead.
- Reading a 60% "probability to beat" as a win. It is a coin flip with a slight lean.
In AB Test WP
AB Test WP runs A/B and A/B/n tests inside WordPress: variants are matched by a CSS class or created point-and-click on the live page, visitors are assigned in the browser, and results are stored in your own database. Free shows the counts and the leader; Pro adds the significance test, a Bayesian probability-to-beat and an uplift interval.
Related: Check a result in the significance calculator · How to A/B test in WordPress, step by step · all glossary terms.
Common questions
Is A/B testing the same as split testing?
Yes. Split testing is another name for the same method. In practice "split test" often refers to a URL split test, where the two versions are separate pages and visitors are redirected, while an A/B test usually changes an element inside one page.
How many visitors does an A/B test need?
It depends on your baseline conversion rate and the smallest improvement you want to detect. Detecting a 20% relative lift on a 5% baseline needs about 8,156 visitors per variant at 95% confidence and 80% power; a 10% lift needs almost four times that. The sample-size calculator gives the number for your figures.
Can I A/B test with low traffic?
Yes, but test larger changes so the effect is big enough to detect, run for longer, and choose a goal that happens often, such as a click, rather than a rare purchase.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.