Type I and Type II errors

A Type I error occurs when a hypothesis test rejects a true null hypothesis, concluding an effect exists when it does not, also called a false positive. A Type II error is the opposite: failing to reject a null hypothesis that is actually false, missing a real effect, also called a false negative.

The probability of a Type I error is controlled directly by the significance threshold chosen before the test, conventionally called alpha and usually set to 0.05; this is the same threshold a p-value is compared against. The probability of a Type II error is called beta, and 1 - beta is statistical power, the test's ability to detect a real effect when one exists. The two error types trade off against each other: making a test more conservative to avoid false positives increases the chance of missing real effects, and vice versa.

Both matter because the cost of each error differs by context: a false positive in a medical screening test causes unnecessary worry and follow-up, while a false negative delays needed treatment. The common pitfall is running many hypothesis tests without adjusting the threshold, which inflates the overall Type I error rate, the exact issue addressed by the multiple comparisons problem.