T-test

A t-test evaluates whether the difference between the means of two groups, or between a group mean and a fixed value, is larger than would be expected from random sampling variation alone. It is one of the most common tools for analyzing A/B testing results on continuous metrics like revenue per user or time on page.

The test produces a t-statistic, roughly the observed difference divided by an estimate of its variability, which is then converted into a p-value using the t-distribution, a bell-shaped curve similar to the normal distribution but with heavier tails to account for the extra uncertainty of estimating variance from a sample. Variants include the independent-samples t-test for two separate groups, the paired t-test for before-and-after measurements on the same subjects, and Welch's t-test, which does not assume the two groups have equal variance.

T-tests matter because they give a standardized way to judge whether a difference is likely more than noise. They differ from a chi-square test, which is used for categorical counts rather than means of continuous data. The main pitfalls are applying a t-test to heavily skewed data with a small sample, where the normality assumption breaks down, and running many t-tests across metrics without correcting for the multiple comparisons problem.