How solid is any of this?
544 entriesOf the 544 cognitive biases in this index, 164 have been retested and found again, 42 were retested and not found, 249 give mixed results, and 89 have no located replication at all.
Every entry in this index carries a replication verdict, and the verdicts are quoted from the papers rather than assessed here. This page is what they add up to.
The finding
Where the original study and the replication both report an effect on the same scale, the replication is the smaller of the two in 63 cases out of 85. That is the replication crisis stated as a measurement rather than as an argument: not that these effects are fictional, but that the first published number is usually the largest one anybody gets. 22 of the 85 pairs held up or came out larger.
The median gap between a claim being published and somebody retesting it is 28 years, across the 455 entries where a replication has been located.
How the index breaks down
| Verdict | Entries | Share | What it means |
|---|---|---|---|
| Replicated | 164 | 30% | A repeat found the effect again. |
| Mixed | 249 | 46% | Repeats disagree, or the effect holds in some conditions and not others. |
| Failed to replicate | 42 | 8% | A repeat looked and did not find it. |
| No replication located | 89 | 16% | No replication attempt was found. That is a fact about the literature, not a verdict on the effect. |
The 25 most looked-up, with their verdicts
Entries are numbered in the order they were written, and the order was set by how often each bias is searched for. These twenty-five are therefore the best-known in the index, and they are where a general claim about cognitive bias is usually coming from.
| № | Bias | Field | Verdict |
|---|---|---|---|
| 001 | Sunk cost | Decision and judgement | Replicated |
| 002 | Ego depletion | Social and self | Failed to replicate |
| 003 | Facial feedback | Perception | Mixed |
| 004 | Dunning-Kruger effect | Belief and probability | Mixed |
| 005 | Confirmation bias | Belief and probability | Replicated |
| 006 | Anchoring effect | Decision and judgement | Replicated |
| 007 | Framing effect | Decision and judgement | Replicated |
| 008 | Survivorship bias | Belief and probability | Mixed |
| 009 | Pygmalion effect | Social and self | Mixed |
| 010 | False consensus effect | Social and self | Replicated |
| 011 | Fundamental attribution error | Social and self | Replicated |
| 012 | Affect heuristic | Decision and judgement | Mixed |
| 013 | Barnum effect | Belief and probability | Replicated |
| 014 | Macbeth effect | Social and self | Failed to replicate |
| 015 | Side-effect effect | Social and self | Replicated |
| 016 | Choosing versus rejecting | Decision and judgement | Failed to replicate |
| 017 | Tempting fate | Belief and probability | Mixed |
| 018 | Question order effect | Belief and probability | Mixed |
| 019 | Moral typecasting | Social and self | Replicated |
| 020 | Disfluency effect | Belief and probability | Failed to replicate |
| 021 | Mere-exposure effect | Perception | Mixed |
| 022 | Spotlight effect | Social and self | No replication located |
| 023 | Halo effect | Social and self | Replicated |
| 024 | Cognitive dissonance | Social and self | Mixed |
| 025 | Gambler's fallacy | Belief and probability | Mixed |
Other ways to cut this
Each verdict above has its own page. Beyond them: every entry where both effect sizes are reported on the same scale, the decade each effect was named in and how each decade has held up, the people named on four or more entries, and every other name these effects go by.
Where this could be wrong
The verdicts are only as good as the replications that exist, and replication effort is not spread evenly: famous effects get retested and obscure ones do not, so a verdict of "no replication located" is partly a statement about attention. The 249 mixed verdicts cover a wide range, from an effect that survives in one culture and not another to one that survives only with the original materials. And an entry is a summary of a literature, which means the judgement of what counts as the replication of record has been made by one person and can be argued with. Each entry names its own sources so that argument can start from the same documents.