Illusion of validity
Confidence in a prediction tracks how neatly the evidence fits a story, not how well it predicts, so consistent and redundant inputs produce the most assured judgements.
Confidence in a prediction tracks how neatly the evidence fits a story, not how well it predicts, so consistent and redundant inputs produce the most assured judgements.
The claim separates two things that feel like one. How sure a forecaster should be depends on how much the evidence can tell him, which is a property of its validity. How sure he is depends on how well the evidence composes into a picture of the outcome, which is a property of the fit. Those come apart most sharply when inputs are redundant.
Several correlated scores say close to what one score says, and they say it without contradicting each other, so the picture is clean while the information added is small. Kahneman and Tversky state the paradox on page 249 of the founding paper: high intercorrelations among inputs increase confidence and decrease validity. The illusion is named for its persistence, and the paper hedges that persistence. It often survives recognition of its illusory character, and the case they give is the selection interview.
Replicated The confidence half of the effect repeated cleanly in a preregistered sample of 791, and nobody has replicated the accuracy half because the original never measured it.
Chan and Feldman registered a replication of all seven studies in Kahneman and Tversky (1973), and Peer Community in Registered Reports granted in-principle acceptance on 31 May 2024. Data collection had already begun during the later part of Stage 1 review. The recommender judged that the design did not change after that point and awarded the top level of bias control anyway.
They recruited 801 US participants on Prolific against a target of 800. The illusion-of-validity demonstration came out as the original did. Ten people gave impossible grade averages and were dropped, leaving 791. In the experimental group of 397, confidence was higher predicting from correlated inputs, 62.47 against 55.99 on a 0 to 100 scale, t(396) = 8.30, p < .001, d = 0.42, 95% CI [0.31, 0.52].
A control group of 394 saw the same two test pairs with the labels swapped and gave 66.57 against 56.42, d = 0.63, 95% CI [0.52, 0.74]. The original counterbalanced the labels in the same way, so the switch is a repeat rather than an addition. What the replication cannot do is show the confidence was misplaced, since the original design has no correct answer to score against.
The report is candid that higher confidence is not by itself evidence that people were judging by representativeness. Its verdict on the 1973 paper as a whole is mixed, and entry no. 145 carries that record. The regression problem moved furthest from the original, and entry no. 248 carries its figures.
Three referees from one officeillustration
A hiring panel reads three references that agree closely. All three writers sat in the same office and watched the same work, so the second and third add little the first did not carry. The agreement makes the panel sure, and the picture it is sure about rests on one vantage point.
The tidy pupilillustration
Every report on a pupil praises neat books, good attendance and effort, and nothing in the file contradicts anything else. A teacher predicts top grades with real confidence. The file is consistent because the same habits were graded six times, not because it forecasts examinations.
Five indicators off one feedillustration
An analyst lines up five measures that all point the same way and writes a firm forecast. Each measure is built from the same weekly sales file, so they cannot disagree. Confidence has risen with the neatness of the picture while the information behind it has not.
Cases marked as illustrations describe nobody in particular. They are there to make the pattern recognisable, not to report that a particular thing happened.
The founding paper's own demonstration sits in that section, and it measures confidence alone. Subjects predicted a grade point average from two pairs of aptitude tests, one pair called highly correlated and the other uncorrelated, and rated their confidence in each. Half of them saw the two labels reversed. All the tests were said to predict college performance equally well.
Confidence was higher for the correlated pair across the entire range of predicted scores, t = 4.80, df = 129, p < .001. The paper prints no sample size for that demonstration. There is no accuracy criterion in the design either, because the students are hypothetical and no true grade exists. The general argument is stated beside it: given inputs of stated validity, the multiple correlation with the criterion falls as the inputs correlate with each other, and that same correlation is what makes a profile look consistent.
The paper names the applied case, saying an interviewer feels great confidence despite knowing that interviews are notoriously fallible. Kahneman's own account of that case is autobiographical. Assessing officer candidates for the Israeli army in the mid-1950s, he watched groups of eight men, insignia of rank removed, lift a telephone pole and cross a 2.5-metre wall without the pole touching the ground or the wall.
He and a colleague felt they could see who would lead. Feedback from the officer-training school arrived about monthly and put the accuracy of those ratings at negligible, and the next day's candidates produced the same sense of insight. He also reports that a structured interview schedule he then designed raised validity, with correlations of close to .30 against about .10 for the previous method, and he gives both as recollection.
The nearest thing to a systematic accuracy record is the clinical-versus-mechanical literature. Grove and colleagues meta-analysed 136 studies of health and behaviour in which a judge and a formula worked from the same predictors, with a weighted mean effect of 0.086 on their transformed scale, SD 0.12, median 0.080. Under their rubric, 63 studies of the 136 favoured the formula by a pragmatic margin, 65 were level, and 8 favoured the clinician.
The variable that mattered most is the one the illusion predicts: where a clinical interview was among the predictors the mean effect against the judge was 0.224, against 0.070 where it was not, t(134) = 5.02, p < .0001. Ægisdóttir and colleagues, reviewing 56 years of mental-health prediction, drew 92 effect sizes from 67 studies and report a weighted mean of -0.15 favouring the formula.
Their most stringent subset, 41 studies and 48 effect sizes, gives -0.12. Their much-quoted 13% is not a share of anything measured: it converts that subset into a binomial display in which clinical decisions are right 47% of the time and statistical ones 53%. The dissociation has also been produced directly. Hall, Ariss and Todorov had participants predict National Basketball Association games from win records and half-time scores, with half also given the team names.
Names left accuracy flat or lowered it while raising the confidence of fans, who in one experiment reached 6.29 on a 0 to 10 scale against 5.23 without names. In a betting version with 20 participants and feedback after every game, those given names earned $1.64 against $2.78, t(19) = 2.35, p < .03, and most still believed the names had helped.
Outside the laboratory, Ben-David, Graham and Harvey collected more than 13,300 one-year market-return forecasts from chief financial officers over ten years and found realised returns inside the executives' own 80% intervals 36% of the time.
8 primary research2 replication and reanalysis2 commentary and critique
The term entered print in "On the psychology of prediction", 80(4), 237-251, as the heading of its section "Confidence and the illusion of validity" on pages 248 and 249. Kahneman says he coined it years earlier, as a second lieutenant in the Israel Defence Forces psychology branch, and that it reached the technical literature almost twenty years later. The same argument appears as point five of Tversky and Kahneman's review "Judgment under uncertainty: Heuristics and biases", issued in August 1973 as an Oregon Research Institute report for the Office of Naval Research and published in Science the following year.
The claim, and the replication this entry reads its figures from.
The 1973 paper was read in full. Unpaywall records it as closed, OpenAlex lists no open location, PsycNet served no text and Semantic Scholar holds no record under its DOI. One of the two CiteSeerX links OpenAlex carries is dead, but the Wayback Machine still replays the file it served, and that scan is the published article at its journal pagination.
So the demonstration, its t of 4.80 and the passage that names the effect are read here from the article rather than from the replication's tables. The article prints no sample size for that demonstration, which means the 130 given for it elsewhere is its degrees of freedom plus one, and no figure of that kind appears above.
Tversky and Kahneman's report version was read in the scan at the Internet Archive, and the Science article in the copy at the University of California, Irvine. The report's own sentence on redundancy says that it increases accuracy, which contradicts the argument around it; Science prints the corrected form, that redundancy decreases accuracy even as it increases confidence.
Einhorn and Hogarth quote page 249 without the word 'often', which the original uses to hedge how far the illusion persists. The replication is a Stage 1 registered report whose results exist as an undergraduate thesis deposited on OSF. No Stage 2 was found on 21 September 2026 at Crossref, at Peer Community in Registered Reports, in the OSF project or by open-web search, and the project's files have not changed since 31 May 2024.
That thesis contradicts itself once, giving df = 396 in its text and 399 in its summary table for the same test. Written before the decision, it also says its level of bias control was cut from six to two because data came first, which the published recommendation does not bear out. No effect size is printed for the original, because no source consulted supplies one, and computing one here from t and df would be arithmetic rather than reading.
Kahneman's army figures cannot be checked: the .30 and .10 correlations are his own recollection in a memoir written decades later, and no data from the unit was published. Ægisdóttir and colleagues were read in full in the copy at gwern.net, the publisher having answered a plain request with a 403. Törngren and Montgomery's study of stock-market professionals is still unread, so nothing from it appears above.
Taylor and Francis answers with a Cloudflare challenge rather than a paywall notice, ResearchGate answers with a captcha, Unpaywall lists no repository copy and Semantic Scholar records it as closed with no file.
The story everyone tells about the illusion of validity is the officer-selection story, and it is not the evidence for it. Kahneman coined the phrase there and published no data from it; the correlations he quotes are remembered, and the experiment that carries the name is a grade-prediction task run on students. The second error runs the other way.
Study 6 shows that consistent inputs raise confidence, and shows nothing about whether they lower accuracy, because the predicted students do not exist. Treating the replication as proof that confident forecasters are wrong reads a result out of a design that could not produce it. The clinical-versus-mechanical literature is also flattened in the retelling. Grove and colleagues did not find that judges are useless: half of their 136 studies were level and 8 favoured the clinician, and the finding with teeth is narrower, that the gap widens when an interview is one of the predictors.
Finally, the illusion is not the same as general overconfidence. It is a claim about which predictions attract the most confidence, and its prediction is specific: the redundant, tidy, story-shaped case, rather than the ambiguous one.
Every claim on this page was held against these sources on 2026-09-21. Nothing here is written from memory. Written and checked by Krishna Chagti; how entries are written and corrected. Found a mistake? Report it, with the sentence and a source.