Glossary

Statistical significance

A threshold judgment that an observed effect is unlikely to be due to chance alone, given the data and test used.

Also called: significance testing, statistically significant

Statistical significance is a threshold-based judgment: a result is called "significant" when the observed effect is large enough, relative to the noise in the data, that it would be unlikely to occur if there were truly no effect. It is a property of a specific test and a chosen threshold (commonly 5%), not an inherent quality of the finding itself.

In practice, a test yields a p-value, which is compared to a pre-chosen significance level; if the p-value falls below that level, the result is declared significant. This depends heavily on sample size: with enough data, even a trivially small and practically meaningless difference can become statistically significant, while a real and important effect can fail to reach significance in a small or noisy sample.

Statistical significance is used across A/B testing, regression analysis, and any comparison between groups to decide whether an observed difference warrants acting on. The most damaging misreading is treating "statistically significant" as "important" or "true"—significance says nothing about the size of an effect or its practical value, and a significant result can still be a false positive, especially when many comparisons are tested at once without adjusting the threshold.

Last reviewed September 19, 2026

In the index now

Related terms

P-value The probability of observing data at least as extreme as the result, if the null hypothesis were actually true. Confidence interval A range calculated from sample data that, under repeated sampling, would contain the true value a stated percentage of the time. Sample size The number of observations in a study or test, which determines how precisely and reliably an effect can be measured. A/B testing A controlled experiment that compares two variants to measure which one produces a better outcome. Regression analysis A statistical method for estimating the relationship between one outcome variable and one or more predictor variables.

Related tools

AB Tasty French web experimentation and personalization platform defaulting to Bayesian reporting, now merged with VWO under the Wingify brand. Adalysis PPC auditing and split-testing tool that automatically checks Google and Microsoft Ads accounts against best-practice rules. AP VoteCast Survey-based electorate research product from the AP and NORC that replaces exit polling for characterizing who voted and why. Convert Experiences Privacy-focused web A/B testing platform that re-analyzes the same experiment under fixed-horizon frequentist, sequential frequentist or Bayesian statistics. Datadog Experiments (formerly Eppo) Warehouse-native experimentation platform known for its CUPED implementation, acquired by Datadog in 2025 and rebranded Datadog Experiments. EViews Econometric software for time-series modeling, forecasting, and panel-data analysis, widely used by economists and financial analysts.

Related guides

How to analyze survey data Weighting, nonresponse, open-ends and testing differences across groups — the steps between a raw response file and a defensible finding. How to choose a survey and voice-of-customer tool Survey and VoC tools split into self-serve builders and enterprise XM suites; pick by distribution channel, text-analysis depth and who owns the program. How to choose an experimentation and feature-flag platform Pick by where your experiment data comes from — flag telemetry or your own warehouse — and which statistical engine your team trusts, not feature count. How to choose statistical software The decisions that matter are code vs menus, general-purpose vs domain specialist, and whether your field already has a de facto standard you can't ignore. How to read an A/B test result Significance, power, confidence intervals, sample ratio mismatch and peeking — what a dashboard's green checkmark does and does not tell you. How to set up an experimentation program From your first A/B test to a durable program — the plumbing, the statistics rules, the review ritual, and the tools that support each stage.