Chi-square test

A chi-square test evaluates whether the counts observed across categories, such as how many users clicked a button in each of three page variants, differ from the counts that would be expected if there were no real relationship, or if a variable followed a specified distribution. It is built for categorical, count-based data rather than the continuous, numeric data a t-test handles.

The test statistic sums, across all categories, the squared difference between observed and expected counts divided by the expected count: sum((observed - expected)^2 / expected). Larger deviations from what chance would produce push the statistic higher, which then maps to a p-value via the chi-square distribution. Two common forms are the test of independence, checking whether two categorical variables are related, and the goodness-of-fit test, checking whether observed proportions match an assumed distribution.

Chi-square tests are widely used to validate experiment setups, including a sample ratio mismatch check that confirms traffic was actually split into the intended proportions between test variants. The main pitfall is applying the test when expected counts in some categories are very small, conventionally below 5, which makes the chi-square approximation unreliable and calls for an exact test instead.