Glossary

A/B testing

A controlled experiment that compares two variants to measure which one produces a better outcome.

Also called: split testing, AB test, A/B test

A/B testing is a controlled experiment that randomly assigns users or units to two (or more) variants—typically a control and a change—and compares an outcome metric between groups. Random assignment is what makes it a form of causal inference: because the groups differ only by chance and by the variant they saw, any consistent difference in outcomes can be attributed to the change itself, not to who happened to see it.

Running a valid test requires deciding the sample size and test duration in advance, based on the minimum effect size worth detecting, and then checking the result against a pre-registered threshold for statistical significance rather than peeking at the data repeatedly and stopping once a result looks favorable. A p-value or confidence interval on the difference between groups is the standard output.

A/B testing is widely used to evaluate product changes, pricing, messaging, and page design because it isolates cause and effect better than observing before-and-after trends. Common pitfalls include stopping early when results look good ("peeking"), running too many simultaneous tests without correcting for false positives, and treating a statistically significant but practically tiny effect as a meaningful win.

Last reviewed September 19, 2026

In the index now

Related terms

Statistical significance A threshold judgment that an observed effect is unlikely to be due to chance alone, given the data and test used. P-value The probability of observing data at least as extreme as the result, if the null hypothesis were actually true. Sample size The number of observations in a study or test, which determines how precisely and reliably an effect can be measured. Causal inference The set of methods used to determine whether one variable actually causes a change in another, not merely correlates with it. Incrementality testing An experiment measuring the true causal lift of a marketing activity by comparing it against a held-out control. Confidence interval A range calculated from sample data that, under repeated sampling, would contain the true value a stated percentage of the time.

Related tools

AB Tasty French web experimentation and personalization platform defaulting to Bayesian reporting, now merged with VWO under the Wingify brand. Adalysis PPC auditing and split-testing tool that automatically checks Google and Microsoft Ads accounts against best-practice rules. Adapty Subscription infrastructure and analytics for mobile apps, with a no-code paywall builder and A/B testing, priced as a RevenueCat alternative. Amplitude Cloud-based digital analytics platform for event tracking, behavioral cohorts, session replay, and feature experimentation. AppTweak ASO and app store market-intelligence platform for keyword research, competitor benchmarking, and creative testing. Bloomreach Discovery AI-driven ecommerce search and merchandising product whose real-time personalization draws on Bloomreach's customer data engine.

Related guides

How to choose a game analytics platform Game analytics splits between standalone, engine-neutral tools and analytics bundled into a backend-as-a-service — pick based on what else you need. How to choose a personalization engine Most tools sold as personalization engines are search and merchandising platforms with a personalization layer on top — know which job you're buying. How to choose a product analytics tool Product analytics tools split by how events get captured, who reads them, and whether in-app engagement is bundled in — decide those first. How to choose a session replay tool Pick a session replay tool by who watches the recordings — engineers debugging or marketers optimizing — and by what a free tier actually costs you. How to choose an email marketing platform Pick an email and lifecycle platform by what triggers a send, whether it proves lift with a holdout, and how pricing scales with your list. How to choose an experimentation and feature-flag platform Pick by where your experiment data comes from — flag telemetry or your own warehouse — and which statistical engine your team trusts, not feature count.