Multiple comparisons problem

The multiple comparisons problem is the statistical fact that running many hypothesis tests increases the chance that at least one will come back "significant" purely by chance, even if no real effect exists anywhere. At a standard 0.05 significance threshold, testing 20 unrelated hypotheses will, on average, produce one false positive even when nothing real is going on.

This happens because each individual test carries its own small, accepted risk of a Type I error, and those risks compound across tests: the more comparisons made, more metrics checked in an A/B test, more segments sliced, more variants compared, the higher the overall chance that something looks significant by luck alone. Corrections such as the Bonferroni adjustment, which divides the significance threshold by the number of tests, or the less conservative Benjamini-Hochberg procedure, are designed to control this inflated error rate.

The problem matters anywhere teams check many metrics or subgroups and then highlight whichever one crossed the significance threshold, a pattern sometimes called p-hacking when done, even unintentionally, after the fact. The safeguard is to decide which metrics and segments matter before looking at results, apply a correction when testing many at once, and treat any post hoc "significant" finding among many checked as a hypothesis to test again, not a conclusion.