Incrementality testing

Incrementality testing measures the actual causal effect of a marketing activity by comparing outcomes for a group exposed to it against a comparable group that was not. It is an application of causal inference and A/B testing principles to advertising and marketing spend, rather than to a product feature.

Common designs include geo-holdout tests, where an ad campaign runs in some regions but not matched control regions, and ghost or public-service ads, where the control group sees a blank or unrelated placement instead of the tested ad. The difference in outcomes between exposed and control groups isolates the lift the marketing activity actually caused, as opposed to conversions that would have happened anyway.

This matters because attribution modeling and even marketing mix modeling can credit a channel for conversions it did not cause—someone who would have bought regardless still gets attributed to the last ad they saw. Incrementality tests correct for that, at the cost of needing enough scale and traffic for a statistically reliable result. Their biggest pitfall is generalization: a lift measured in one geography, audience, or season doesn't automatically apply to a different channel mix or time period.