Sampling bias

Sampling bias occurs when the way a sample is drawn systematically over- or under-represents parts of the population, so the sample does not reflect the group it is meant to describe. It is a flaw in the sampling method itself, such as an incomplete sampling frame, convenience sampling, or a survey distributed through a channel that reaches some groups more than others.

A classic example is a phone survey conducted only during business hours, which systematically excludes people who work standard daytime jobs and overrepresents retirees, students, and shift workers. Because the bias is baked into who had a chance of being selected, it does not shrink with a larger sample size, and it is not captured by a margin of error, which only accounts for random variation, not this kind of systematic skew.

Sampling bias matters because it can quietly invalidate survey results, experiment rollouts, and any analysis that assumes a sample represents a broader population. It overlaps closely with selection bias and nonresponse bias, where people who chose not to respond may differ systematically from those who did. The fix is a sampling design that gives every relevant group a known, non-zero chance of inclusion, or survey weighting applied afterward to correct known imbalances.