A/A test

An A/A test runs the same experimentation pipeline as an A/B test but assigns users to two groups that receive identical treatment. Because there is no real difference between the groups, any statistically significant result the test reports is, by definition, a false positive produced by the tooling, not a real effect.

Teams run A/A tests to validate an experimentation platform before trusting it with real decisions: randomization should split users roughly evenly, the reported metrics for the two "identical" groups should track closely, and a significance test repeated across many A/A pairs should flag a false positive at roughly the nominal rate, about 5% of the time at a 0.05 threshold. Systematic deviation from this, many more or fewer false positives than expected, or unequal group sizes, points to bugs in randomization, logging, or metric computation rather than bad luck.

A/A tests are also a practical way to catch a sample ratio mismatch before it contaminates real experiments, and to estimate a metric's natural variance for later sample-size planning. They are diagnostic exercises, not a substitute for peer review of the underlying randomization and instrumentation code.