Holdout Test

A holdout test deliberately withholds a marketing activity from a comparable subset of an audience (the control group) to measure its true causal effect against a group that did receive it (the test group).

The entire validity of a holdout test rests on the two groups being genuinely comparable before the test starts — randomized assignment at the user level, or matched geographic regions with similar historical performance for geo-based tests, are the two standard ways to achieve this. A control group that differs systematically from the test group (different demographics, different prior behavior) will produce a misleading lift number regardless of how careful the rest of the analysis is.

Platform-native tools — Meta Conversion Lift, Google's geo experiments — automate a lot of the mechanics, but the underlying logic is identical to a manual geo-split holdout.