Sample Size

Sample size is the number of users, sessions, or conversions included in a test — and it directly determines how much you can trust the result.

Small sample sizes produce noisy, unreliable results even when the underlying math is done correctly — a 26% lift measured across 400 users per group carries far more uncertainty than the same 26% lift measured across 40,000 users per group, even though both might report an identical headline number. Sample size calculators (used before a test starts) estimate how many users are needed to reliably detect a given effect size at a chosen significance threshold.

Low-traffic pages, new products, or niche audiences often simply don't generate enough volume to reach significance in a reasonable timeframe — in that case, the honest answer is that a rigorous quantitative test isn't currently feasible, not that the test should be run anyway and trusted.