Glossary
A/B test hypothesis: definition and template
The sentence that turns a change into an experiment, a template for writing it, and examples that show the difference between a hunch and a hypothesis.
Definition: An A/B test hypothesis is a testable statement of what you will change, what you expect to happen, and why: "Because visitors say the pricing is confusing, adding a comparison table will raise trial sign-ups." It fixes the metric and the reasoning before the test, so the result teaches you something whichever way it goes.
The template
Because [observation], changing [element] from [A] to [B] will [raise / lower] [metric], because [reason].
Every bracket does a job. The observation grounds the test in evidence. The element and the change define the variant. The metric is the conversion goal you will measure, named now so it cannot be swapped later. The reason is the mechanism you believe in, which is what you learn about when the result comes in.
Where hypotheses come from
Analytics that show where visitors leave; session recordings and heatmaps that show what they hesitate over; support tickets and sales calls that repeat the same objection; surveys and user tests; and competitors who solved the same problem differently. A hypothesis born from one of these has a reason attached; one born from a meeting usually does not.
Good and weak examples
- Weak: "A green button will convert better." No observation, no reason, no metric.
- Better: "Because recordings show visitors scrolling past the grey button, changing it to the brand's accent colour will raise clicks on Start trial, because it will be noticed."
- Weak: "A shorter form will help." Help what?
- Better: "Because 40% of visitors abandon the form at the phone field, removing it will raise completed sign-ups, because the field signals a sales call."
One hypothesis, one test
A test with two hypotheses cannot tell you which one the result supports. If two changes must ship together, test the pair as one variant and write the hypothesis about the pair. Record the outcome either way: a rejected hypothesis with a clear reason is a learning, and the next test starts from it.
A worked example
Observation: the pricing page's exit rate is twice the site average, and support tickets ask which plan includes a feature. Hypothesis: because visitors cannot tell the plans apart, adding a feature comparison table under the plan cards will raise trial sign-ups from the pricing page, because the decision becomes visible. Metric: trial sign-ups, counted once per visitor. Sample: 8,156 per variant at a 5% baseline and a 20% MDE. Result, either way, says something about whether plan clarity was the problem.
Common mistakes
- Writing the hypothesis after the test, to fit the result.
- Naming no metric, then choosing the one that happened to move.
- Bundling three changes and one hypothesis.
- Discarding a rejected hypothesis instead of recording what it ruled out.
Related: Size the test before you start · What to A/B test first · all glossary terms.
Common questions
Do I need a hypothesis to run a test?
The tool will run without one. You need one to learn anything from the result: without a stated metric and reason, a win teaches you nothing you can reuse and a loss teaches you less.
What if the result contradicts the hypothesis?
That is a result. Write down what it rules out and what it suggests instead. Many of the most useful tests are the ones that killed a confident assumption.
How specific should it be?
Specific enough that two people would build the same variant and measure the same number from it. If it could describe several different tests, it is not finished.
Run the numbers on your own test
Free calculators, no email wall: significance, sample size, conversion rate and SRM.