A/B testing glossary

A/B test

Showing two versions of a page or element to different visitors at the same time and measuring which one achieves a goal more often. It is the basic unit of experimentation; every other term on this page qualifies it.

Full explanation: What is A/B testing?

Go deeper: How to A/B test in WordPress

A/B/n test

An A/B test with more than two versions, all compared against the same control. Each extra variant needs its own full sample, so the traffic a test needs grows with the number of arms.

Full explanation: What is A/B/n testing?

Go deeper: How many visitors a test needs

Anti-flicker snippet

A small piece of code, loaded before the page renders, that hides content until the testing script has applied the variant. Cloud testing tools add one to every page, and it is a common source of slower loads.

Go deeper: Caching and CDN compatibility

Assignment (bucketing)

How a visitor is placed into a variant. Cookieless tools keep the assignment in the browser's local storage or derive it from a random identifier, so the same person keeps seeing the same version on later visits.

Go deeper: Cookieless testing and GDPR

Audience targeting

Limiting who takes part in a test, for example by device, by new versus returning visitor, or by a URL parameter. A narrower audience gives a cleaner answer about that group and takes proportionally longer to fill its sample.

Go deeper: Targeting in AB Test WP

Baseline conversion rate

The control's conversion rate before the test starts. It is one of the three inputs to every sample-size calculation, and the lower it is, the more visitors a test needs.

Go deeper: Sample-size calculator

Bayesian probability to beat

The probability, under a Bayesian model, that a variant is better than the control. It is easier to read than a p-value, but it is a different number answering a different question, and the two are not interchangeable.

Go deeper: A/B testing statistics made simple

Canonical URL

The address a page declares as its authoritative version. In a split URL test the variant page points its canonical at the original, so search engines index one page and the test does not compete with itself.

Go deeper: SEO settings for URL tests

Confidence interval

The range within which the true uplift plausibly lies, given the data so far. A wide interval means the test has not yet pinned the effect down, even when the point estimate looks like a win.

Full explanation: What is a confidence interval?

Go deeper: Significance calculator

Confidence level

One minus the false-positive rate you accept. At 95% confidence you accept that one in twenty tests of a change that does nothing will still look like a winner by chance.

Go deeper: What 95% confidence means

Control

The current version, also called A or the baseline. Every variant is measured against it, and it is the version you keep if nothing beats it.

Full explanation: Control vs variant in an A/B test

Conversion

A visitor completing the action the test measures: reaching a page, clicking an element or link, submitting a form, or buying. What counts as a conversion is fixed by the goal before the test starts.

Full explanation: What is a conversion in marketing?

Conversion rate

Conversions divided by visitors, expressed as a percentage. It is the number a test compares between variants, and the quantity every calculator on this site works with.

Full explanation: Conversion rate: definition and formula

Go deeper: Conversion rate calculator

Cookieless testing

Running a test without setting cookies, typically by keeping the assignment in browser local storage. It avoids cookie mechanics but does not by itself place a test outside privacy law, which covers local storage too.

Go deeper: Cookieless testing and GDPR

CSS class (selector)

The label a testing tool uses to find the element under test, such as a class added to a section in a page builder. Class-based tools work in any builder or theme that lets you add a class to an element.

Go deeper: How class-based testing works

Element test

A test that swaps one element on a page, such as a headline, a button or a whole section, while everything else stays the same. It gives a clean answer about that one change.

Go deeper: How it works

False negative (Type II error)

Missing a real improvement. Statistical power controls how often this happens: at the usual 80% power, one real effect in five goes undetected.

False positive (Type I error)

Declaring a winner when the variants are really the same. The confidence level caps how often this happens, but only if you read the result once, at the planned sample.

Go deeper: Why a winner did not hold

Flicker

The visible switch from the original content to the variant while a testing script runs, also called a flash of original content. Flicker-free tools keep the tested element hidden until the variant is applied, so the original never flashes, with a failsafe timeout.

Go deeper: Caching, CDNs and flicker

Frequentist and Bayesian statistics

Two schools of statistics. Frequentist methods such as z-tests and p-values ask how rare the data would be if there were no effect; Bayesian methods estimate the probability of the effect directly. Well-run tests reach the same decisions either way.

Full explanation: Frequentist vs Bayesian A/B testing

Go deeper: Statistics made simple

Goal

The specific conversion a test counts, chosen before the test starts. One goal per test keeps the answer clean; a test with several goals usually ends with the one that happened to look best.

Go deeper: Goals in AB Test WP

Hypothesis

A falsifiable statement of what you expect and why, in the form "changing X to Y will increase Z". Writing one turns a hunch into a test that teaches something whichever way the result goes.

Full explanation: A/B test hypothesis: definition and template

Go deeper: Your first A/B test

Impression

A visitor being shown a variant. Impressions are the denominator of the conversion rate, so whether a tool counts them per visitor or per page view changes what its rates mean.

Learnings

What a test taught you, regardless of its outcome. A recorded hypothesis and result keep a team from re-testing the same idea a year later.

Go deeper: Picking what to test

Minimum detectable effect (MDE)

The smallest uplift a test is designed to catch, chosen before it starts. It sets the sample size more than anything else: halving the MDE roughly quadruples the visitors needed.

Full explanation: Minimum detectable effect (MDE)

Go deeper: How many visitors a test needs

Multivariate test (MVT)

Testing combinations of several elements at once to learn how they interact. Every combination is its own arm, so an MVT needs far more traffic than an A/B test of the same elements one at a time.

Full explanation: What is multivariate testing?

Novelty effect

A temporary lift caused by something being new rather than better. It fades as returning visitors get used to the change, which is one reason tests run for whole weeks rather than days.

Go deeper: Six ways a winner vanishes

p-value

The probability of seeing a gap at least as large as the observed one if the variants were actually identical. A p-value under 0.05 corresponds to 95% confidence.

Full explanation: What is a p-value?

Go deeper: Significance calculator

Page cache and CDN compatibility

Whether a test still works when pages are served from a cache or a content delivery network instead of being generated fresh. Tools that assign variants in the browser survive caching; server-side assignment usually does not.

Go deeper: Caching and CDN compatibility

Peeking

Checking a running test and stopping the moment it looks significant. Because confidence swings widely early on, peeking produces far more false winners than the 5% the confidence level promises.

Go deeper: How long to run a test

Sample ratio mismatch (SRM)

When the observed split between variants differs from the planned one by more than chance allows, for example 58/42 instead of 50/50. It is a sign that the test is broken and that its results cannot be trusted.

Full explanation: What is sample ratio mismatch (SRM)?

Go deeper: SRM checker

Sample size

The number of visitors per variant a test needs before its result can be trusted, calculated from the baseline rate, the minimum detectable effect, the confidence level and the power. Decided before the test starts, never after.

Go deeper: Sample-size calculator

Seasonality

Regular changes in visitor behaviour by day of the week, time of month, or marketing calendar. Tests run in whole weekly cycles so both arms see the same mix of visitors.

Go deeper: Planning test duration

Segmentation

Breaking results down by a visitor attribute, such as device, after the test. Useful for seeing where an effect comes from, and dangerous when used to hunt for a segment that happens to look significant.

Split testing

Another name for A/B testing. In practice it often means a URL split test, where the versions are separate pages and visitors are redirected, rather than an element test inside one page.

Full explanation: Split testing vs A/B testing

Split URL test (redirect test)

A test between two separate pages at different addresses. Visitors are sent to one or the other and kept there on return visits. Used when variants differ in too many ways to swap single elements.

Go deeper: How to split test two landing pages

Statistical power

The chance a test detects a real effect of the size you specified as the minimum detectable effect. 80% power is the convention; lower power means more real winners go unnoticed.

Go deeper: Power and sample size

Statistical significance

A result is significant when chance alone would rarely produce a gap as large as the one observed. At the usual 95% level, rarely means less than one time in twenty. Significance says a difference is probably real, not that it is large.

Full explanation: Statistical significance in A/B testing

Go deeper: Significance calculator

Traffic allocation (weighting)

The share of visitors sent to each variant. An even split reaches significance fastest; uneven weights limit how many people see a risky variant, at the cost of a longer test.

Two-proportion z-test

The standard frequentist test for comparing two conversion rates. It is the formula behind most A/B significance and sample-size calculators, including the ones on this site.

Full explanation: The two-proportion z-test

Go deeper: Sample-size calculator

Uplift (lift)

The improvement of a variant over the control. A move from 3.0% to 3.6% is a 20% relative uplift and a 0.6 percentage-point absolute one; most tools and this site quote the relative figure.

Full explanation: Uplift (lift) in A/B testing

Go deeper: Conversion rate and uplift calculator

Variant (challenger)

Any version other than the control: B in a two-way test, and B, C, D and so on in an A/B/n test. The variant is what you are proposing to ship if it wins.

Winner

The variant declared better at the end of a test. Shipping it means making it the permanent page; a winner that does not hold up in production usually points to a sizing or peeking problem.

Go deeper: Why your test said +40% and nothing changed