Statistical Significance

Statistical significance is a measure of how likely an observed difference between two groups is to be real, rather than a result of random chance.

The conventional threshold in most marketing testing is a p-value below 0.05 — meaning there's less than a 5% chance the observed difference would occur if there were truly no underlying effect. This threshold is a widely-used convention, not a law of nature; some teams use stricter (0.01) or looser (0.10) bars depending on how costly a false positive would be.

Significance depends heavily on sample size. A real, meaningful effect can fail to reach significance simply because too few people were in the test; conversely, with a large enough sample, even a trivially small and practically meaningless difference can become "statistically significant." Significance tells you whether an effect is likely real — it doesn't by itself tell you whether the effect is big enough to matter for the business.