What is A/B/n testing?

Definition: An A/B/n test compares more than two versions at once: the control plus two or more variants, each shown to a random share of visitors and measured on the same goal. It answers a wider question in one run, but every extra variant needs its own full sample, so the test takes proportionally longer.

When to use it

When you have several genuinely different ideas for the same element and no reason to prefer one: four headline angles, three pricing layouts. Testing them in one run avoids the sequential problem where the second test starts on a different week, with different traffic, than the first.

The traffic maths

The sample size is calculated per variant, then multiplied by the number of arms. A test that needs 8,156 visitors per variant needs 16,312 in total with two arms, 32,624 with four and 48,936 with six. At 1,000 visitors a day that is 17 days, 33 days and 49 days. The sample-size calculator takes the number of variants as an input.

The multiple-comparisons trap

Each comparison against the control carries the agreed false-positive rate. Five variants at 95% confidence give five chances of a false positive, roughly a 23% chance that at least one variant "wins" by luck. Either raise the confidence threshold as arms are added (a Bonferroni-style correction divides 5% by the number of comparisons) or treat a single narrow winner among many losers with suspicion and confirm it in a follow-up A/B test.

A/B/n is not multivariate

A/B/n changes one element in several ways. A multivariate test changes several elements and tests the combinations. The distinction matters because multivariate tests multiply the traffic requirement much faster.

A worked example

A sign-up page converts at 5%. You want to detect a 20% relative lift with three headline variants plus the control. Per variant: 8,156 visitors. Total: 32,624. At 1,000 visitors a day the test needs 33 days, so plan for five whole weeks. If only two headlines are serious candidates, drop the third and finish two weeks sooner.

Common mistakes

• Adding a fourth variant "to see what happens" and doubling the test's duration.
• Comparing variants to each other after the fact and picking the best pair.
• Declaring the top arm the winner when it is significant against one variant but not against the control.
• Splitting traffic unevenly without a reason; equal splits reach the sample soonest.

In AB Test WP

AB Test WP Free runs two-variant tests; Pro runs up to six variants per test, and the built-in planner multiplies the per-variant sample by the number of variants so the total and the days are shown before the test starts.

Related: Sample-size calculator with a variants setting · How many visitors an A/B test needs · all glossary terms.