What is sample ratio mismatch (SRM)?

Definition: Sample ratio mismatch (SRM) is when the share of visitors in each variant differs from the split you configured by more than chance allows: 5,400 against 4,600 on a 50/50 test, for example. It means something is diverting or dropping visitors unevenly, and until the cause is found the conversion comparison cannot be trusted.

How it is detected

A chi-square goodness-of-fit test compares the observed counts with the counts the configured split would produce. Because the test is a safety check rather than a discovery, practitioners use a strict threshold, commonly a p-value below 0.001 or 0.01, so that only real imbalances trigger it. The SRM checker runs the test from the visitor counts and the intended split.

χ² = Σ (observed - expected)² / expected

Common causes

• The redirect of a URL split test fails for some browsers or is blocked by a cache, so one variant is under-counted.
• One variant's tracking call fails, for instance because a script error stops it recording impressions.
• Bot traffic is filtered on one variant and not the other.
• A targeting or scheduling rule applies to one variant only.
• The variant loads slower and impatient visitors leave before the impression is recorded.

What to do about it

Stop reading the results, find the cause, fix it and restart the test with fresh counts. Do not try to repair the data by removing visitors after the fact; you cannot know which of them the bug affected. A mismatch does not tell you which variant is really better; it tells you the test cannot say.

A worked example

A 50/50 test records 5,400 visitors in the control and 4,600 in the variant. Expected: 5,000 each. χ² = 400²/5,000 + 400²/5,000 = 64, p far below 0.001: SRM. A split of 5,080 to 4,920 gives χ² = 2.56, p = 0.11, which is ordinary random variation. The first test is broken; the second is fine.

Common mistakes

• Ignoring a mismatch because the winning variant is the one you hoped for.
• Using a 5% threshold and flagging healthy tests every week.
• Excluding visitors after the fact to make the ratio look right.
• Checking the ratio at the end only; a check while the test runs saves weeks.

In AB Test WP

AB Test WP Pro checks for sample ratio mismatch on running tests and will not declare a winner while a warning is active. The free SRM checker on this site runs the same chi-square test on any tool's numbers.

Related: SRM checker · Why an A/B test winner did not hold · all glossary terms.