Open data · CC BY 4.0 what each one claims, and what replicated no ads · no tracking
Bias Atlascognitive biases

Selection bias

also known as — Sample selection bias · Selection effect · Ascertainment bias
Selection bias is the gap between the quantity a study is after in the population and the same quantity in the subset that ended up available for analysis.
Does it replicate? No replication located No replication attempt was found, and none is the right thing to look for; what exists instead is a record of the bias measured in particular datasets. No replication attempt has been located, which is a statement about the literature rather than about the effect.
VerdictNo replication located
First published1979
FieldBelief and probability
Sources18
Last checked2026-08-17
What it claims

What does Selection bias mean?

This is a property of a sample rather than a habit of a mind, which it shares with survivorship bias, entry 8, and Berkson's paradox, entry 58, and with almost nothing else in this collection. Something decided who got into the data, and the estimate computed inside that subset answers a different question from the one asked. Hernán states the general form in his 2017 commentary as the bias that arises when the parameter of interest in a population differs from the parameter in the subset of individuals available for analysis.

The structural account came earlier. Hernán, Hernández-Díaz and Robins describe the shared causal structure as "conditioning on a common effect of 2 variables, one of which is either exposure or a cause of exposure and the other is either the outcome or a cause of the outcome". They set that against confounding, which they define as "the existence of common causes of exposure and outcome".

A common effect and a common cause are different objects and they call for opposite handling. Heckman reached a version of the problem from econometrics. He wrote in 1979 that it "may arise in practice for two reasons": self selection by the people being studied, and "sample selection decisions by analysts or data processors", which he said "operate in much the same fashion as self selection".

Verdict

No replication attempt was found. That is a fact about the literature, not a verdict on the effect. The other 88.

The field
57 of the 192 entries in belief and probability share this verdict.
Does it replicate?

Has Selection bias been retested?

No replication located No replication attempt was found, and none is the right thing to look for; what exists instead is a record of the bias measured in particular datasets.

No replication attempt was found for this effect. That is not evidence that it fails: the search came up empty, and an absence in the literature says nothing either way. How this index searches.

Examples

What are some examples of Selection bias?

The survey sent to current customersillustration

A firm emails a satisfaction survey to everyone on its active list and reports a score of four out of five. Everyone who gave up on the product left the list first, so the people whose answers would have moved the number are the ones the sampling frame removed.

The consultation nobody young answeredillustration

A council posts a questionnaire about a park and receives nine hundred replies, most from people who attend residents' meetings on weekday evenings. The result is a genuine measurement of something. It is not the measurement of local opinion that the covering letter says it is.

The volunteers at the health fairillustration

A study recruits its participants from people who came to a free screening day. Turning up already goes with caring about one's health and with having a free morning, and both of those go with the outcome being measured, so the estimate answers a question about attenders.

Cases marked as illustrations describe nobody in particular. They are there to make the pattern recognisable, not to report that a particular thing happened.

The experiments

What experiments is Selection bias based on?

The design

Heckman's paper treats the bias "that results from using nonrandomly selected samples to estimate behavioral relationships as an ordinary specification error or 'omitted variables' bias", and offers "a simple consistent two stage estimator". His examples are labour-market ones. Market wages are observed for working women whose market wage exceeds their home wage at zero hours of work, and for union members who found the nonunion alternative less desirable.

The occupational literature supplies measured cases. Burns, Bodner, Jammer, Collins and Swaen calculated standardised mortality ratios for 114,683 US chemical industry employees who worked at least three days between 1960 and 2005. All-cause mortality came out at 79, 95% CI 78 to 80, against the general population, with 70 for non-malignant respiratory disease, 95% CI 67 to 73.

Employed people are not a random draw of the living, and the paper opens by noting that occupational studies typically observe a 20% deficit in overall mortality. Chen and Seaton examined 85 occupational cohorts of workers exposed to organic solvents. Their abstract gives a weighted all-cause standardised mortality ratio of 89.5, 95% CI 89.0 to 90.2, and attributes that figure to their own earlier work rather than to this paper.

What this paper adds is a regression, which finds that the higher the rates of loss to follow up, the greater the effect, p = 0.04. Psychology's version is about who volunteers. Henrich, Heine and Norenzayan, reporting an analysis by Arnett of the top journals in six subdisciplines from 2003 to 2007, write that "68% of subjects came from the United States, and a full 96% of subjects were from Western industrialized countries", and that this means "96% of psychological samples come from countries with only 12% of the world's population".

In the Journal of Personality and Social Psychology, 67% of the American samples and 80% of the samples from other countries were composed solely of undergraduates in psychology courses. Their summary is that "a randomly selected American undergraduate is more than 4,000 times more likely to be a research participant than is a randomly selected person from outside of the West".

Survey research has the best-documented failures because the truth arrives on election night. AAPOR's task force on the 2020 US general election found the average error on the margin, among polls in the final two weeks, was 4.5 points for national polls and 5.1 points for state-level presidential polls, with the signed error too favourable to Biden by 3.9 and 4.3 points.

The task force ruled out, as primary sources of the error, late deciders, failure to weight by education, the demographic composition of the electorate, differential turnout among Democratic and Republican respondents, and reluctance to name Trump to an interviewer. What it could not rule out was who declined to answer at all: "At least some of the polling error in 2020 was caused by unit nonresponse." The 2024 task force repeated the exercise and found smaller errors.

Across 611 polls fielded in the final two weeks, the average absolute error on the two-party margin was 3.3 points, against 5.3 in 2020 and 5.2 in 2016. National presidential polls missed by 2.6 points and state-level presidential polls by 3.0. That report rebuilt the historical dataset, so its own figures for earlier years differ from the ones the 2020 task force published.

15 primary research3 commentary and critique

Origin

Who first described Selection bias, and when?

Year1979
WhoJames J. Heckman
WhereEconometrica

"Sample Selection Bias as a Specification Error", Econometrica 47(1), 153 to 161. The term was already in circulation. Heckman's own reference list carries his 1976 paper on the common structure of truncation and sample selection models, Gronau's 1974 "Wage Comparisons: A Selectivity Bias" and Lewis's reply of the same year, and his footnote 2 credits Gronau and Lewis with a grouped data version of the estimation method.

What 1979 established is the treatment of the problem as an ordinary specification error with a usable correction. The structural definition that epidemiology now works from arrived 25 years later, with Hernán, Hernández-Díaz and Robins in 2004. Berkson's 1946 hospital-admission case, entry 58 here, is an earlier documented species.

Where it runs out

When does Selection bias not apply?

Selection bias is not an experimental effect. There is no protocol to re-run, no effect size that travels between labs and no confidence interval that would settle anything. Given a selection process connected to what is being estimated, the selected sample is distorted, and that follows from the structure rather than being tested. Every number above is a measurement of one dataset.

What is genuinely contested is the scope of the structural account, and the argument has moved twice since 2004. Hernán published a commentary in 2017 arguing that "selection bias may arise when conditioning on a variable C that is not a collider". He read that off Greenland's 1977 example, in which the causal risk ratio was 1.69 in the whole population and 2.05 among the uncensored, with about 28% of people censored, and censoring associated with the outcome at a risk ratio of 1.36 but with the exposure at 1.02.

Lu, Gonsalves and Westreich pushed the other way in 2024. They argue that including a collider in a regression should not count as selection bias, because "there is no selection" and the sample analysed does not change, and that the case belongs under overadjustment bias instead. Lu, Howe, Zivich, Gonsalves and Westreich traced the same argument through the literature in 2025.

The size of the distortion is contested too, and it depends on the mechanism rather than on the amount of missing data. Kristman, Manno and Côté simulated 1,000 replications of a cohort of 500 observations and found no important bias at losses to follow-up from 5% to 60% where the loss was missing completely at random or missing at random, and seriously biased odds ratios at low levels of loss where it was missing not at random.

The correction is not free either. Puhani surveyed Monte Carlo studies of Heckman's two-step estimator and concluded that where collinearity is absent "the full-information maximum likelihood estimator is preferable to the limited-information two-step method", and that where collinearity is present, subsample ordinary least squares or the two-part model is the most robust of the estimators that are simple to calculate.

Commonly misread as

How is Selection bias commonly misread?

The first error is a translation error between fields. Hernán records that econometrics and statistics have historically used selection bias for something "essentially equivalent to the epidemiologic term confounding", and Lu, Howe, Zivich, Gonsalves and Westreich write that in clinical trials and economics the term "is commonly used to refer to what epidemiologists define as confounding". Heckman's censored wage sample and Hernán's collider are not the same object, and a reader crossing between the two literatures has to check which one a paper means.

The second error is the one that follows from the first, because the remedies point in opposite directions. A confounder is handled by adjusting for it. A collider is turned into a bias by adjusting for it, so the reflex that fixes one causes the other. Berkson's paradox, entry 58, is the case where that goes wrong inside a hospital register, and it is one species of selection bias rather than a synonym for it.

Survivorship bias, entry 8, is another species, with survival as the filter and the cases that did not last long enough to be counted as the ones missing from the data. The third error is reading a low response rate as a measure of bias. The 1936 Literary Digest poll is the standing example of a poll destroyed by who replied, and its response rate was not the small number the retelling implies.

Lohr and Brick record ten million ballots mailed, almost 2.4 million returned and a response rate of 24 per cent, which is "much higher than the response rate in many of today's polls". The two papers that examined it most closely differ by a point on its headline number: Lohr and Brick report the poll as predicting 54 per cent of the popular vote for Landon, and Squire, printing the magazine's final count of 1,293,669 for Landon, 972,897 for Roosevelt and 83,610 for Lemke, puts Landon's share at 55 per cent.

Roosevelt took more than 60 per cent of the vote and 523 electoral votes. Squire went to a Gallup survey of May 1937 that asked people whether they had received a Literary Digest ballot and whether they had returned it. Among the 780 who said they received one, Roosevelt led 55 to 44; among the 493 who said they returned it, Landon led 51 to 48.

His rough decomposition puts about 11 points of the error on the sample and about 7 on the responses, and he notes that the Gallup poll overstates the winner by 5 points and that too many people claimed to have returned a ballot. Lohr and Brick show that reweighting the magazine's published state-by-state returns by respondents' reported 1932 vote moves ten states, including California and New York, and turns an unweighted prediction of 16 states and 161 of 531 electoral votes for Roosevelt into 26 states and 276, the right winner from the wrong sample.

Their verdict on the magazine is that its real failure was not the response rate it published but that it "failed to assess the uncertainty this implied about their estimates". The fourth error is treating the label as a finding. Naming a selection process establishes that an association might have been manufactured. It does not give the direction or the size, and both still have to be argued for from the data.

Sources

Sources for Selection bias

  1. 1Heckman, J. J. (1979). Sample Selection Bias as a Specification Error. Econometrica, 47(1), 153-161. Full text read from a JSTOR scan hosted at Arizona State University; the quotations are from the abstract and the opening two pages, and the reference list and footnote 2 are the source of the note on prior use of the term. The jstor.org page for the article returns a login wall.primary
  2. 2Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A Structural Approach to Selection Bias. Epidemiology, 15(5), 615-625. Abstract read from the PubMed E-utilities XML record, PMID 15308962. The publisher's full text at journals.lww.com returned HTTP 403 behind a browser challenge on 17 August 2026, and Unpaywall and Europe PMC list no open copy, so the full text was not obtained.primary
  3. 3Hernán, M. A. (2017). Invited Commentary: Selection Bias Without Colliders. American Journal of Epidemiology, 185(11), 1048-1050. Full text read from the Europe PMC rendering of PMC6664806, which reproduces the published typesetting; the entry's earlier note that the full text was not obtained was wrong. The Greenland 1977 figures quoted here are Hernán's description of that paper, not read in Greenland.primary
  4. 4Lu, H., Gonsalves, G. S., & Westreich, D. (2024). Selection Bias Requires Selection: The Case of Collider Stratification Bias. American Journal of Epidemiology, 193(3), 407-409. Full text read from the Europe PMC rendering of PMC10911840.primary
  5. 5Lu, H., Howe, C. J., Zivich, P. N., Gonsalves, G. S., & Westreich, D. (2025). The evolution of selection bias in the recent epidemiologic literature: a selective overview. American Journal of Epidemiology, 194(3), 580-584. Full text read from the Europe PMC rendering of PMC11879605. The printed subtitle is separated by an em dash rather than a colon.primary
  6. 6Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61-83. Full text read from the author's PDF. The sampling percentages quoted here are Henrich and colleagues reporting Arnett (2008), not their own count.primary
  7. 7Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. American Psychologist, 63(7), 602-614. The source of the 68%, 96%, 12% and undergraduate figures. Crossref confirms the record; the paper itself was not opened, and the figures are quoted here as Henrich and colleagues report them. Thalmayer and colleagues describe the same 2008 count differently, as over 70% of samples and authors from a country holding 5% of the world's population, which is a further reason not to treat the retold percentages as Arnett's own wording.secondary
  8. 8Burns, C. J., Bodner, K. M., Jammer, B. L., Collins, J. J., & Swaen, G. M. H. (2011). The healthy worker effect in US chemical industry workers. Occupational Medicine, 61(1), 40-44. Full structured abstract read from the PubMed E-utilities XML record, PMID 21078828.primary
  9. 9Chen, R., & Seaton, A. (1996). The Influence of Study Characteristics on the Healthy Worker Effect: A Multiple Regression Analysis. Occupational Medicine, 46(5), 345-350. Full abstract read from the PubMed E-utilities XML record, PMID 8918148. Unpaywall lists the Oxford Academic PDF as free, and that URL returned HTTP 403 on 17 August 2026, so the full text was not obtained.primary
  10. 10Kristman, V., Manno, M., & Côté, P. (2004). Loss to Follow-Up in Cohort Studies: How Much is Too Much? European Journal of Epidemiology, 19(8), 751-760. Full abstract read from the PubMed E-utilities XML record, PMID 15469032; Unpaywall lists no open copy and the full text was not obtained.primary
  11. 11Clinton, J., et al. (2021). Task Force on 2020 Pre-Election Polling: An Evaluation of the 2020 General Election Polls. American Association for Public Opinion Research. Josh Clinton of Vanderbilt University chaired. Full report read; figures taken from the executive summary, and the list of ruled-out explanations from the section headed Factors That Do Not Explain the Polling Error, which rules them out as primary sources rather than absolutely.primary
  12. 12Pasek, J., et al. (2025). Task Force on 2024 Pre-Election Polling: An Evaluation of the 2024 General Election Polls. American Association for Public Opinion Research. Josh Pasek of the University of Michigan chaired. Full report read; figures taken from the executive summary and section 2.2. Its footnote 2 records that the 2024 committee assembled a more comprehensive dataset and adopted new methods, so its accuracy estimates for earlier years differ from the earlier reports, by less than a point except for 2000 national polls.primary
  13. 13Thalmayer, A. G., Toscanelli, C., & Arnett, J. J. (2021). The neglected 95% revisited: Is American psychology becoming less American? American Psychologist, 76(1), 116-129. Full abstract read from the PubMed E-utilities XML record, PMID 32271027. The University of Zurich postprint that Unpaywall lists returned HTTP 404 on 17 August 2026, so the full text was not obtained.primary
  14. 14Kennedy, C., et al. (2017). An Evaluation of 2016 Election Polls in the United States. American Association for Public Opinion Research. Full report read. It gives an average absolute error of 2.2 points for national polls and 5.1 points for 423 state polls, and its Table 10 shows the share of final polls weighting on education, from 18% in Michigan to 52% nationally.primary
  15. 15Lohr, S. L., & Brick, J. M. (2017). Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll. Statistics, Politics and Policy, 8(1), 65-84. Full text read. Their reweighting uses the state-by-state table the magazine published on 31 October 1936, not the individual returns, which no longer exist.primary
  16. 16Squire, P. (1988). Why the 1936 Literary Digest Poll Failed. Public Opinion Quarterly, 52(1), 125-133. Full text read from a scan of the Oxford University Press typesetting; the entry's earlier note that the full text was not obtained was wrong. His conclusion is that the magazine's sample and the response "were both biased and jointly produced the wildly incorrect estimate of the vote". The tables are from American Institute of Public Opinion poll 83, a quota sample, and Squire says so.primary
  17. 17Puhani, P. (2000). The Heckman Correction for Sample Selection and Its Critique. Journal of Economic Surveys, 14(1), 53-68. Abstract read at RePEc, which reproduces the published abstract; Unpaywall lists no open copy and the full text was not obtained. The quoted sentence is from that abstract.secondary
  18. 18Berkson, J. (1946). Limitations of the Application of Fourfold Table Analysis to Hospital Data. Biometrics Bulletin, 2(3), 47. Cited here only to fix the date of the earlier species; see entry 58 for its own sources.secondary

Every claim on this page was held against these sources on 2026-08-17. Nothing here is written from memory. Written and checked by Krishna Chagti; how entries are written and corrected. Found a mistake? Report it, with the sentence and a source.

About Selection bias

Is Selection bias real?

No replication attempt was found, and none is the right thing to look for; what exists instead is a record of the bias measured in particular datasets. No replication attempt has been located, which is a statement about the literature rather than about the effect.