Selection bias

Selection bias occurs when the process used to gather or include data systematically favors certain kinds of observations over others, so the resulting sample does not represent the population it is supposed to describe. Any conclusion drawn from a biased sample can look statistically solid while still being wrong about the broader population.

It shows up in many forms: a survey posted only in a power-user community will overrepresent engaged customers; an analysis that only includes users who completed onboarding excludes everyone who dropped out along the way and so cannot speak to why they left; and survivorship bias, one of its most common variants, only examines entities that "survived" some filter, like companies still operating or accounts still active. It is closely related to but distinct from sampling bias, which specifically concerns a flawed sampling method or sampling frame rather than any selective inclusion process.

Selection bias matters because it cannot generally be fixed after the fact with more sophisticated statistics; the fix has to happen in how the data is collected, by deliberately including the excluded group or clearly scoping conclusions to the population actually sampled. The common pitfall is generalizing a finding from a self-selected or filtered group, like app reviewers or survey respondents, to all users.