Holdout group

A holdout group is a set of users who are intentionally excluded from a treatment, whether a new feature, a marketing campaign, or a retargeting program, so their behavior can serve as the counterfactual baseline for everyone else. Unlike a short-lived A/B test control, a holdout is often kept in place for weeks or months to measure cumulative or long-run impact.

The difference between the holdout's outcome and the treated group's outcome, adjusted for any pre-existing gap between them, is the estimated incremental effect of the treatment. This is the core logic behind incrementality testing and conversion lift studies, and it is what separates a channel's true contribution from activity that would have happened anyway.

Holdouts matter most where correlation is easy to mistake for causation, such as marketing channels that tend to reach users who were already going to convert. A common pitfall is letting the holdout shrink or leak, users gradually finding their way into treatment through other channels or manual overrides, which biases the comparison toward showing no effect. Holdouts also carry a real opportunity cost, since the excluded group is deliberately denied a treatment believed to help, which is why holdout size and duration are usually set as narrowly as the required precision allows.