Open data · CC BY 4.0 what each one claims, and what replicated no ads · no tracking
Bias Atlascognitive biases

Illusion of validity

also known as — Unwarranted confidence in prediction · Illusion of predictive validity
Confidence in a prediction tracks how neatly the evidence fits a story, not how well it predicts, so consistent and redundant inputs produce the most assured judgements.
Does it replicate? Replicated The confidence half of the effect repeated cleanly in a preregistered sample of 791, and nobody has replicated the accuracy half because the original never measured it.
VerdictReplicated
First published1973
FieldBelief and probability
Sources12
Last checked2026-09-21
What it claims

What does Illusion of validity mean?

The claim separates two things that feel like one. How sure a forecaster should be depends on how much the evidence can tell him, which is a property of its validity. How sure he is depends on how well the evidence composes into a picture of the outcome, which is a property of the fit. Those come apart most sharply when inputs are redundant.

Several correlated scores say close to what one score says, and they say it without contradicting each other, so the picture is clean while the information added is small. Kahneman and Tversky state the paradox on page 249 of the founding paper: high intercorrelations among inputs increase confidence and decrease validity. The illusion is named for its persistence, and the paper hedges that persistence. It often survives recognition of its illusory character, and the case they give is the selection interview.

Verdict

A repeat found the effect again. The other 163.

The field
54 of the 192 entries in belief and probability share this verdict.
Does it replicate?

Has Illusion of validity been retested?

Replicated The confidence half of the effect repeated cleanly in a preregistered sample of 791, and nobody has replicated the accuracy half because the original never measured it.

Pooled across the replicationd = 0.42 (95% CI 0.31 to 0.52)
no effect Replication Replication: d = 0.42, 95% CI 0.31 to 0.52
Plotted against the point where there is no effect. Hover a point for its interval.

Chan and Feldman registered a replication of all seven studies in Kahneman and Tversky (1973), and Peer Community in Registered Reports granted in-principle acceptance on 31 May 2024. Data collection had already begun during the later part of Stage 1 review. The recommender judged that the design did not change after that point and awarded the top level of bias control anyway.

They recruited 801 US participants on Prolific against a target of 800. The illusion-of-validity demonstration came out as the original did. Ten people gave impossible grade averages and were dropped, leaving 791. In the experimental group of 397, confidence was higher predicting from correlated inputs, 62.47 against 55.99 on a 0 to 100 scale, t(396) = 8.30, p < .001, d = 0.42, 95% CI [0.31, 0.52].

A control group of 394 saw the same two test pairs with the labels swapped and gave 66.57 against 56.42, d = 0.63, 95% CI [0.52, 0.74]. The original counterbalanced the labels in the same way, so the switch is a repeat rather than an addition. What the replication cannot do is show the confidence was misplaced, since the original design has no correct answer to score against.

The report is candid that higher confidence is not by itself evidence that people were judging by representativeness. Its verdict on the 1973 paper as a whole is mixed, and entry no. 145 carries that record. The regression problem moved furthest from the original, and entry no. 248 carries its figures.

Read off Chan, H. C., & Feldman, G. (2024). Representativeness heuristic in intuitive predictions: Replication Registered Report of problems reviewed in Kahneman and Tversky (1973). Bachelor of Social Sciences thesis, University of Hong Kong, with results, materials and data deposited at OSF project 8zhcj.

Examples

What are some examples of Illusion of validity?

Three referees from one officeillustration

A hiring panel reads three references that agree closely. All three writers sat in the same office and watched the same work, so the second and third add little the first did not carry. The agreement makes the panel sure, and the picture it is sure about rests on one vantage point.

The tidy pupilillustration

Every report on a pupil praises neat books, good attendance and effort, and nothing in the file contradicts anything else. A teacher predicts top grades with real confidence. The file is consistent because the same habits were graded six times, not because it forecasts examinations.

Five indicators off one feedillustration

An analyst lines up five measures that all point the same way and writes a firm forecast. Each measure is built from the same weekly sales file, so they cannot disagree. Confidence has risen with the neatness of the picture while the information behind it has not.

Cases marked as illustrations describe nobody in particular. They are there to make the pattern recognisable, not to report that a particular thing happened.

The experiments

What experiments is Illusion of validity based on?

The design

The founding paper's own demonstration sits in that section, and it measures confidence alone. Subjects predicted a grade point average from two pairs of aptitude tests, one pair called highly correlated and the other uncorrelated, and rated their confidence in each. Half of them saw the two labels reversed. All the tests were said to predict college performance equally well.

Confidence was higher for the correlated pair across the entire range of predicted scores, t = 4.80, df = 129, p < .001. The paper prints no sample size for that demonstration. There is no accuracy criterion in the design either, because the students are hypothetical and no true grade exists. The general argument is stated beside it: given inputs of stated validity, the multiple correlation with the criterion falls as the inputs correlate with each other, and that same correlation is what makes a profile look consistent.

The paper names the applied case, saying an interviewer feels great confidence despite knowing that interviews are notoriously fallible. Kahneman's own account of that case is autobiographical. Assessing officer candidates for the Israeli army in the mid-1950s, he watched groups of eight men, insignia of rank removed, lift a telephone pole and cross a 2.5-metre wall without the pole touching the ground or the wall.

He and a colleague felt they could see who would lead. Feedback from the officer-training school arrived about monthly and put the accuracy of those ratings at negligible, and the next day's candidates produced the same sense of insight. He also reports that a structured interview schedule he then designed raised validity, with correlations of close to .30 against about .10 for the previous method, and he gives both as recollection.

The nearest thing to a systematic accuracy record is the clinical-versus-mechanical literature. Grove and colleagues meta-analysed 136 studies of health and behaviour in which a judge and a formula worked from the same predictors, with a weighted mean effect of 0.086 on their transformed scale, SD 0.12, median 0.080. Under their rubric, 63 studies of the 136 favoured the formula by a pragmatic margin, 65 were level, and 8 favoured the clinician.

The variable that mattered most is the one the illusion predicts: where a clinical interview was among the predictors the mean effect against the judge was 0.224, against 0.070 where it was not, t(134) = 5.02, p < .0001. Ægisdóttir and colleagues, reviewing 56 years of mental-health prediction, drew 92 effect sizes from 67 studies and report a weighted mean of -0.15 favouring the formula.

Their most stringent subset, 41 studies and 48 effect sizes, gives -0.12. Their much-quoted 13% is not a share of anything measured: it converts that subset into a binomial display in which clinical decisions are right 47% of the time and statistical ones 53%. The dissociation has also been produced directly. Hall, Ariss and Todorov had participants predict National Basketball Association games from win records and half-time scores, with half also given the team names.

Names left accuracy flat or lowered it while raising the confidence of fans, who in one experiment reached 6.29 on a 0 to 10 scale against 5.23 without names. In a betting version with 20 participants and feedback after every game, those given names earned $1.64 against $2.78, t(19) = 2.35, p < .03, and most still believed the names had helped.

Outside the laboratory, Ben-David, Graham and Harvey collected more than 13,300 one-year market-return forecasts from chief financial officers over ten years and found realised returns inside the executives' own 80% intervals 36% of the time.

8 primary research2 replication and reanalysis2 commentary and critique

Origin

Who first described Illusion of validity, and when?

Year1973
WhoDaniel Kahneman and Amos Tversky
WherePsychological Review

The term entered print in "On the psychology of prediction", 80(4), 237-251, as the heading of its section "Confidence and the illusion of validity" on pages 248 and 249. Kahneman says he coined it years earlier, as a second lieutenant in the Israel Defence Forces psychology branch, and that it reached the technical literature almost twenty years later. The same argument appears as point five of Tversky and Kahneman's review "Judgment under uncertainty: Heuristics and biases", issued in August 1973 as an Oregon Research Institute report for the Office of Naval Research and published in Science the following year.

The claim, and the replication this entry reads its figures from.

Where it runs out

When does Illusion of validity not apply?

The 1973 paper was read in full. Unpaywall records it as closed, OpenAlex lists no open location, PsycNet served no text and Semantic Scholar holds no record under its DOI. One of the two CiteSeerX links OpenAlex carries is dead, but the Wayback Machine still replays the file it served, and that scan is the published article at its journal pagination.

So the demonstration, its t of 4.80 and the passage that names the effect are read here from the article rather than from the replication's tables. The article prints no sample size for that demonstration, which means the 130 given for it elsewhere is its degrees of freedom plus one, and no figure of that kind appears above.

Tversky and Kahneman's report version was read in the scan at the Internet Archive, and the Science article in the copy at the University of California, Irvine. The report's own sentence on redundancy says that it increases accuracy, which contradicts the argument around it; Science prints the corrected form, that redundancy decreases accuracy even as it increases confidence.

Einhorn and Hogarth quote page 249 without the word 'often', which the original uses to hedge how far the illusion persists. The replication is a Stage 1 registered report whose results exist as an undergraduate thesis deposited on OSF. No Stage 2 was found on 21 September 2026 at Crossref, at Peer Community in Registered Reports, in the OSF project or by open-web search, and the project's files have not changed since 31 May 2024.

That thesis contradicts itself once, giving df = 396 in its text and 399 in its summary table for the same test. Written before the decision, it also says its level of bias control was cut from six to two because data came first, which the published recommendation does not bear out. No effect size is printed for the original, because no source consulted supplies one, and computing one here from t and df would be arithmetic rather than reading.

Kahneman's army figures cannot be checked: the .30 and .10 correlations are his own recollection in a memoir written decades later, and no data from the unit was published. Ægisdóttir and colleagues were read in full in the copy at gwern.net, the publisher having answered a plain request with a 403. Törngren and Montgomery's study of stock-market professionals is still unread, so nothing from it appears above.

Taylor and Francis answers with a Cloudflare challenge rather than a paywall notice, ResearchGate answers with a captcha, Unpaywall lists no repository copy and Semantic Scholar records it as closed with no file.

Commonly misread as

How is Illusion of validity commonly misread?

The story everyone tells about the illusion of validity is the officer-selection story, and it is not the evidence for it. Kahneman coined the phrase there and published no data from it; the correlations he quotes are remembered, and the experiment that carries the name is a grade-prediction task run on students. The second error runs the other way.

Study 6 shows that consistent inputs raise confidence, and shows nothing about whether they lower accuracy, because the predicted students do not exist. Treating the replication as proof that confident forecasters are wrong reads a result out of a design that could not produce it. The clinical-versus-mechanical literature is also flattened in the retelling. Grove and colleagues did not find that judges are useless: half of their 136 studies were level and 8 favoured the clinician, and the finding with teeth is narrower, that the gap widens when an interview is one of the predictors.

Finally, the illusion is not the same as general overconfidence. It is a claim about which predictions attract the most confidence, and its prediction is specific: the redundant, tidy, story-shaped case, rather than the ambiguous one.

Sources

Sources for Illusion of validity

  1. 1Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80(4), 237-251. Read in full on 21 September 2026, from the Wayback Machine's replay of a CiteSeerX copy that OpenAlex lists and that no longer resolves live. The scan carries the journal's own pagination, and the bibliographic record was confirmed at Crossref.primary
  2. 2Tversky, A., & Kahneman, D. (1973). Judgment under Uncertainty: Heuristics and Biases. ONR Technical Report, Oregon Research Institute, August 1973, AD-767 426. Read in full as the scanned report at the Internet Archive. Its section five carries the redundancy argument, with the slip corrected in the journal version below.primary
  3. 3Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124-1131. Read in full in the copy hosted by the University of California, Irvine. It prints the sentence the report garbles, that redundancy decreases accuracy even as it increases confidence.primary
  4. 4Kahneman, D. (2003). Autobiography. In T. Frangsmyr (Ed.), Les Prix Nobel 2002. Read in full at nobelprize.org. The source for the officer-selection account, the coinage of the term, and the remembered validity figures.primary
  5. 5Chan, H. C., & Feldman, G. (2024). Representativeness heuristic in intuitive predictions: Replication Registered Report of problems reviewed in Kahneman and Tversky (1973). University of Hong Kong thesis, deposited at OSF. Read in full; the source for every replication figure above.replication
  6. 6Rahal, R.-M. (2024). The Representativeness Heuristic Revisited: Registered Replication Report of Kahneman and Tversky (1973). Peer Community in Registered Reports. Read in full. The Stage 1 in-principle acceptance of the replication above, posted 29 May 2024 and validated on 31 May, awarding level six bias control.replication
  7. 7Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19-30. Read in full; the source for the 136 studies, the weighted mean and the interview comparison.primary
  8. 8Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., & Rush, J. D. (2006). The Meta-Analysis of Clinical Judgment Project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34(3), 341-382. Read in full in the copy at gwern.net; the publisher answers a plain request with a 403.primary
  9. 9Hall, C. C., Ariss, L., & Todorov, A. (2007). The illusion of knowledge: When more information reduces accuracy and increases confidence. Organizational Behavior and Human Decision Processes, 103(2), 277-290. Read in full; the source for the basketball accuracy, confidence and earnings figures.primary
  10. 10Ben-David, I., Graham, J. R., & Harvey, C. R. (2013). Managerial miscalibration. The Quarterly Journal of Economics, 128(4), 1547-1584. Read in full from the author copy at Duke; the source for the 36% coverage of executives' 80% intervals.primary
  11. 11Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85(5), 395-416. Read in full. It quotes page 249 of the 1973 paper at length, dropping the original's 'often', and models why outcome feedback on accepted cases sustains confidence rather than correcting it.commentary
  12. 12Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515-526. Read in full; the source for Kahneman's later gloss on the illusion as overconfidence in low-validity environments.commentary

Every claim on this page was held against these sources on 2026-09-21. Nothing here is written from memory. Written and checked by Krishna Chagti; how entries are written and corrected. Found a mistake? Report it, with the sentence and a source.

About Illusion of validity

Is Illusion of validity real?

The confidence half of the effect repeated cleanly in a preregistered sample of 791, and nobody has replicated the accuracy half because the original never measured it. The effect sizes from the original study and from the replication are printed side by side above, so the comparison is visible rather than asserted.