P-value

A p-value is the probability, calculated under an assumed null hypothesis (typically "no effect" or "no difference"), of observing a result at least as extreme as the one actually measured. It is a statement about the data given the null hypothesis—it is not the probability that the null hypothesis is true, and it is not the probability that the observed effect is real.

A small p-value means the observed data would be unusual if the null hypothesis were correct, which is treated as evidence against it. Researchers compare the p-value to a pre-set threshold, often 0.05, to declare statistical significance. The p-value depends on both the size of the effect and the sample size: a tiny effect can produce a small p-value if the sample is large enough.

P-values are used throughout A/B testing, regression output, and any hypothesis test to summarize evidence against a default assumption. The two most common misreadings are treating the p-value as "the probability the null hypothesis is true" (it assumes the null is true and asks about the data, not the reverse), and treating a p-value just above or below 0.05 as a meaningful bright line rather than one point on a continuous scale of evidence. A p-value also says nothing about effect size or practical importance—pair it with a confidence interval or effect estimate.