False discovery rate (FDR)

The false discovery rate is the expected fraction of statistically significant results, among all the results flagged as significant, that are actually false positives. It becomes relevant whenever many hypotheses are tested at once, dozens of metrics in one experiment, many subgroup cuts, many simultaneous experiments, because running enough independent tests at a standard 5% significance threshold guarantees that some will turn up significant purely by chance.

This differs from controlling the p-value threshold on a single test, which bounds the chance of one false positive, not the rate across many. The Benjamini-Hochberg procedure is the standard method for controlling FDR: p-values from all the tests are ranked, and each is compared against a threshold that rises with its rank, which is more forgiving than a flat significance cutoff applied to every test but still caps the expected share of false discoveries at a chosen level, commonly 5% or 10%.

FDR control matters anywhere a team checks many metrics or subgroups and needs to avoid mistaking noise for a real finding, a direct concern for guardrail metrics and heterogeneous treatment effects analysis alike. The common pitfall is treating an uncorrected significance threshold as reliable when dozens of comparisons were actually run, or applying an FDR correction only to the metrics that failed to reach significance without it.