Bayesian inference

Bayesian inference treats probability as a degree of belief that gets updated as new data arrives, rather than a fixed long-run frequency. It starts with a prior distribution representing what is believed before seeing the data, combines it with the likelihood of the observed data, and produces a posterior distribution representing updated belief, following Bayes' theorem: posterior is proportional to likelihood times prior.

This differs from classical, frequentist hypothesis testing, which asks how likely the data would be if a fixed null hypothesis were true and produces a p-value, not a probability about the hypothesis itself. A Bayesian result, by contrast, can directly state something like "there is an 85% probability that variant B has a higher conversion rate," which many practitioners find more intuitive, at the cost of needing to specify a prior, a choice that is itself debatable and can influence the result, especially with limited data.

Bayesian methods are increasingly used in A/B testing platforms, forecasting, and any setting where evidence accumulates sequentially and decisions need to be made before a fixed sample size is reached, since Bayesian estimates can be inspected at any time without the inflation of false positives that early peeking causes in frequentist tests. The main pitfall is choosing an uninformative prior carelessly, or an informative one without justifying it, which can quietly bias conclusions.