A/B Test

An A/B test compares two variants (A and B) of something — an ad, a landing page, an email subject line — by randomly showing each to a portion of the audience and measuring which performs better.

A/B testing and holdout testing are close cousins but usually answer different questions. A/B tests typically compare two active variants ("does headline A or headline B convert better"), while holdout tests compare something against nothing ("does this campaign work at all, versus no campaign"). Both require statistical significance checks before acting on results, and both require a large enough sample size to trust the outcome.

The most common A/B testing mistake is stopping the test the moment it looks significant, rather than pre-committing to a sample size or duration — checking results repeatedly and stopping at the first favorable-looking moment inflates the false positive rate substantially.