Statistical significance is a measure of how likely an observed difference between two groups is to be real, rather than a result of random chance.
The conventional threshold in most marketing testing is a p-value below 0.05, meaning there's less than a 5% chance the observed difference would occur if there were truly no underlying effect. This threshold is a widely-used convention, not a law of nature; some teams use stricter (0.01) or looser (0.10) bars depending on how costly a false positive would be.
Significance depends heavily on sample size. A real, meaningful effect can fail to reach significance simply because too few people were in the test; conversely, with a large enough sample, even a trivially small and practically meaningless difference can become "statistically significant." Significance tells you whether an effect is likely real. It doesn't by itself tell you whether the effect is big enough to matter for the business.
Go deeper
Related terms