Statistical power

Statistical power is the probability that a test will detect a real effect when one genuinely exists, rather than missing it. It is defined as 1 - beta, where beta is the probability of a Type II error, see Type I and Type II errors, so higher power means a lower chance of a false negative.

Power depends on several factors that interact: a larger sample size increases power, a larger true effect size is easier to detect and so increases power, a stricter significance threshold, a smaller alpha, reduces power, and less noisy, lower-variance data increases power. Before running a study, researchers commonly perform a power analysis to estimate the sample size needed to reliably detect a minimum detectable effect of a chosen size, typically targeting 80% power as a convention.

Power matters because an underpowered test is not simply inconclusive, it is unlikely to detect a real effect even when one is present, wasting the effort of running it at all. The common pitfall is running an experiment for a fixed, arbitrary duration or sample size without a power calculation, then interpreting a non-significant result as proof of no effect, when the study may simply never have had a realistic chance of finding one.