Glossary
What is multivariate testing?
Testing combinations instead of single changes: what it can reveal, what it costs in traffic, and why most sites get further without it.
Definition: A multivariate test (MVT) changes several elements at once and tests the combinations, so a headline with two versions and a button with three makes six variants. It can reveal interactions between elements, but each combination needs its own sample, which is why MVT suits only high-traffic pages. Most sites get further with a sequence of A/B tests.
How it differs from A/B and A/B/n
An A/B test changes one thing. An A/B/n test changes one thing in several ways. A multivariate test changes several things and runs every combination, so it can tell you not just which headline and which button are best but whether a particular headline works better with a particular button. That interaction is the only thing MVT can measure that a sequence of A/B tests cannot.
The traffic problem
The number of combinations multiplies: three elements with two versions each is eight variants; add a third version of one element and it is twelve. Each combination needs the full per-variant sample, so a test that would take two weeks as an A/B test takes eight to twelve weeks as an MVT on the same traffic. Fractional designs reduce the count by dropping some combinations, at the cost of measuring fewer interactions.
When it is worth it
On pages with tens of thousands of visitors a week, where several elements are in play and you suspect they interact, for example a hero image that only works with a matching headline. Otherwise, run the elements as A/B tests one after another, shipping each winner, which answers the practical question in a fraction of the traffic.
A worked example
A landing page with 2,000 visitors a day wants to test two headlines, two hero images and two button labels. As an MVT that is eight combinations; at 8,156 visitors per arm the test needs 65,248 visitors, about 33 days. As three sequential A/B tests, each needs 16,312 visitors, nine days apiece, so all three finish in under four weeks, and each result is clean.
Common mistakes
- Launching an MVT on a page that cannot feed eight arms in a reasonable time.
- Reading the best combination as a winner when it beat only some of the others.
- Changing elements that plainly do not interact and paying the combinatorial cost anyway.
In AB Test WP
AB Test WP runs A/B and A/B/n tests (up to six variants in Pro). Multivariate testing is being researched; it is not available.
Related: Sample-size calculator · What to A/B test first · all glossary terms.
Common questions
What is the difference between A/B testing and multivariate testing?
A/B testing changes one element; multivariate testing changes several and tests every combination to find interactions. MVT needs far more traffic.
How much traffic does a multivariate test need?
The per-variant sample times the number of combinations. Eight combinations at 8,156 visitors each is more than 65,000 visitors, before allowing for whole weeks.
Can I test two elements without MVT?
Yes. Test them one after another as A/B tests, or combine the two changes into a single variant and test that against the control if you only need to know whether the pair helps.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.