Sample size

Sample size is the number of observations—users, transactions, survey respondents—included in a study or test. It is one of the biggest levers on how precisely and reliably an effect can be estimated: larger samples produce narrower confidence intervals and more reliable p-values, because random noise averages out more as more data accumulates.

Before running a test, researchers typically calculate the minimum sample size needed to reliably detect an effect of a given size, based on the effect worth detecting, the variability in the metric, and the desired confidence level. This calculation matters because a test that is stopped too early, or run on too few observations, is underpowered: it may fail to detect a real effect, or produce an estimate too noisy to trust even if it happens to cross a significance threshold.

Sample size is central to designing valid A/B testing and any other empirical causal inference study—it should be set in advance, not adjusted retroactively based on how the results look partway through. Common pitfalls include running underpowered tests that cannot detect a real, meaningful effect; treating a small sample's significant result with the same confidence as a large sample's; and increasing sample size purely to force a marginal or trivial effect over a significance threshold, mistaking statistical significance for practical importance in the process.