Glossary
Control vs variant in an A/B test
The baseline and the challenger: what each is, why the control runs at the same time rather than being last month's numbers, and how to split traffic between them.
Definition: The control is the existing version of a page or element in an A/B test, the baseline every variant is compared against. A variant is the changed version. Visitors are split between them at random, and the difference in results is credited to the change, because the control holds everything else constant.
Why the control runs at the same time
The obvious shortcut is to compare the new version with last month's numbers. It fails because last month had different traffic, different campaigns, different weather. A control that runs alongside the variant, on the same visitors split at random, absorbs all of that. The only difference left is the change.
A/A tests
An A/A test splits visitors between two identical versions. It should show no significant difference; if it does, something in the setup is broken, such as uneven assignment or a tracking fault. Teams run one when they adopt a new tool or suspect a sample ratio mismatch.
How much traffic the control should get
An even split reaches the required sample soonest, because the test is limited by its smallest arm. Weighting the control higher (say 80/20) limits the exposure of a risky variant at the cost of a longer test; it does not make the comparison less valid as long as the assignment stays random and the tool accounts for the weights.
Naming
The control is often called A, the original or the baseline; variants are B, C and so on. When a variant wins and ships, it becomes the new control for the next test.
A worked example
A checkout page (the control) converts at 3.0%. A variant with trust badges above the pay button is tested against it on a 50/50 split for three whole weeks. Both versions see the same mix of weekday and weekend shoppers, the same campaign traffic and the same stock levels, so when the variant converts at 3.6% the badges are the credible explanation, subject to the significance test.
Common mistakes
- Editing the control while the test runs.
- Comparing a variant with historical data instead of a live control.
- Giving the variant 90% of traffic to "speed things up"; the small control arm slows the test.
- Forgetting that a shipped winner is now the control, and testing against the old one.
In AB Test WP
In AB Test WP the original element or page is the control. Traffic weighting is a Pro feature; Free splits evenly, and each visitor stays on the version they first saw.
Related: Check the split with the SRM checker · How to A/B test in WordPress, step by step · all glossary terms.
Common questions
Can the control change during a test?
No. Any edit to the control changes what the variant is being compared against and invalidates the comparison. Finish or stop the test first.
What is an A/A test?
A test between two identical versions, run to confirm the setup produces no difference on its own. A significant result in an A/A test means the tool or the tracking has a problem.
Should the control get 50% of the traffic?
For the fastest result, yes. Give it more only when the variant is risky enough that you want to limit how many visitors see it, and accept the longer test.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.