Hypothesis testing

Hypothesis testing is a structured procedure for using sample data to decide between two competing claims: a default assumption, the null hypothesis, and an alternative that some effect or difference exists. It underlies A/B testing, scientific research, and most claims of statistical significance.

The general process is to state the null and alternative hypotheses, choose a test suited to the data and question, a t-test, chi-square test, and others each fit different data types, calculate a test statistic from the sample, and convert it into a p-value that measures how surprising the observed data would be if the null hypothesis were true. If the p-value falls below a pre-chosen threshold, typically 0.05, the result is declared statistically significant and the null hypothesis is rejected.

Hypothesis testing matters because it forces a disciplined, pre-specified comparison rather than eyeballing a chart and inventing patterns after the fact. Its two failure modes are formalized as Type I and Type II errors: falsely rejecting a true null, or failing to detect a real effect. Common pitfalls include running many tests without correction, see the multiple comparisons problem, peeking at results early and stopping as soon as they look significant, and treating statistical significance as automatically meaning the effect is large or practically important.