Sequential testing

Sequential testing is a family of statistical methods that let an experiment be monitored and evaluated continuously, or at a series of interim checkpoints, while keeping the overall false-positive rate at the level the analyst set, commonly 5%. It replaces the classical requirement of choosing a sample size in advance and looking at the result exactly once.

Fixed-horizon A/B testing relies on a single p-value computed after data collection stops; checking it early and repeatedly, sometimes called "peeking," inflates the true false-positive rate well above the nominal threshold, because each additional look is another chance to see a fluky significant result. Sequential methods, such as always-valid p-values, sequential probability ratio tests, or group sequential designs with pre-planned interim boundaries, adjust the significance threshold as more looks are taken, so a test can be stopped early for a clear win or loss without inflating error rates.

Sequential testing matters because it lets teams end obviously good or bad experiments sooner rather than waiting out a fixed sample size target, saving traffic and shortening decision cycles. The main pitfall is applying a sequential stopping rule to data that was actually analyzed with fixed-horizon statistics, which reintroduces the peeking problem sequential testing exists to solve.