Open data · CC BY 4.0 what each one claims, and what replicated no ads · no tracking
Bias Atlascognitive biases

How solid is any of this?

544 entries

Of the 544 cognitive biases in this index, 164 have been retested and found again, 42 were retested and not found, 249 give mixed results, and 89 have no located replication at all.

164 replicated249 mixed42 failed89 none located

Every entry in this index carries a replication verdict, and the verdicts are quoted from the papers rather than assessed here. This page is what they add up to.

The finding

Where the original study and the replication both report an effect on the same scale, the replication is the smaller of the two in 63 cases out of 85. That is the replication crisis stated as a measurement rather than as an argument: not that these effects are fictional, but that the first published number is usually the largest one anybody gets. 22 of the 85 pairs held up or came out larger.

The median gap between a claim being published and somebody retesting it is 28 years, across the 455 entries where a replication has been located.

How the index breaks down

Replication verdicts across all 544 entries.
VerdictEntriesShareWhat it means
Replicated 164 30% A repeat found the effect again.
Mixed 249 46% Repeats disagree, or the effect holds in some conditions and not others.
Failed to replicate 42 8% A repeat looked and did not find it.
No replication located 89 16% No replication attempt was found. That is a fact about the literature, not a verdict on the effect.

The 25 most looked-up, with their verdicts

Entries are numbered in the order they were written, and the order was set by how often each bias is searched for. These twenty-five are therefore the best-known in the index, and they are where a general claim about cognitive bias is usually coming from.

The twenty-five most looked-up biases in this index.
№BiasFieldVerdict
001 Sunk cost Decision and judgement Replicated
002 Ego depletion Social and self Failed to replicate
003 Facial feedback Perception Mixed
004 Dunning-Kruger effect Belief and probability Mixed
005 Confirmation bias Belief and probability Replicated
006 Anchoring effect Decision and judgement Replicated
007 Framing effect Decision and judgement Replicated
008 Survivorship bias Belief and probability Mixed
009 Pygmalion effect Social and self Mixed
010 False consensus effect Social and self Replicated
011 Fundamental attribution error Social and self Replicated
012 Affect heuristic Decision and judgement Mixed
013 Barnum effect Belief and probability Replicated
014 Macbeth effect Social and self Failed to replicate
015 Side-effect effect Social and self Replicated
016 Choosing versus rejecting Decision and judgement Failed to replicate
017 Tempting fate Belief and probability Mixed
018 Question order effect Belief and probability Mixed
019 Moral typecasting Social and self Replicated
020 Disfluency effect Belief and probability Failed to replicate
021 Mere-exposure effect Perception Mixed
022 Spotlight effect Social and self No replication located
023 Halo effect Social and self Replicated
024 Cognitive dissonance Social and self Mixed
025 Gambler's fallacy Belief and probability Mixed

Other ways to cut this

Each verdict above has its own page. Beyond them: every entry where both effect sizes are reported on the same scale, the decade each effect was named in and how each decade has held up, the people named on four or more entries, and every other name these effects go by.

Where this could be wrong

The verdicts are only as good as the replications that exist, and replication effort is not spread evenly: famous effects get retested and obscure ones do not, so a verdict of "no replication located" is partly a statement about attention. The 249 mixed verdicts cover a wide range, from an effect that survives in one culture and not another to one that survives only with the original materials. And an entry is a summary of a literature, which means the judgement of what counts as the replication of record has been made by one person and can be argued with. Each entry names its own sources so that argument can start from the same documents.

Questions about these numbers

Do most cognitive biases replicate?

Not straightforwardly. 30% of the 544 entries here record a successful replication and 8% record a failure, but the largest group is the 249 whose results are mixed. The honest summary is that most of these effects are real under some conditions and smaller than first reported.

Which is the most common verdict?

Mixed, at 249 of 544. That is not a hedge: it is what the literature says when an effect survives in some designs, some populations or some measures and not others.