Open data · CC BY 4.0 what each one claims, and what replicated no ads · no tracking
Bias Atlascognitive biases

Response bias

also known as — Response style · Acquiescence bias · Acquiescence response style · Yea-saying · Extreme response style · Survey satisficing · Agreement bias
Response styles are said to move survey answers on their own: some people agree with any assertion, some prefer the endpoints, and some take the first answer that will do.
Does it replicate? Replicated A preregistered replication found acquiescence bias again on both panels it used, but the estimate on the higher-quality panel was about a third the size.
VerdictReplicated
First published1991
FieldBelief and probability
Sources15
Last checked2026-09-21
What it claims

What does Response bias mean?

Several different things travel under this name, and each is measured its own way. Acquiescence is the tendency to endorse an assertion whatever it says. It is measured by asking one question in affirmative and negated form and comparing the two rates, by counting the share of a battery a respondent agrees with, or by comparing correlations between scales worded in the same and in opposite directions.

Extreme responding is the share of answers at the endpoints of a scale, and midpoint responding the share at the middle. Those are counts of scale positions, and they say nothing about which assertion was made. Social desirability is a fourth quantity, scored against validated behaviour rather than against a second wording, and entry 114 covers it. Satisficing is not a measure.

It is Krosnick's proposed mechanism, offered to cover acquiescence, response-order effects and 'don't know' at once. A response criterion in the signal-detection sense, the kind entry 173 reports, is a parameter of a fitted model rather than a survey habit, and nothing here is about it. None of this is a claim about reasoning. It is a claim about instruments, and it is tested by giving randomly chosen halves of a sample different versions of the same question.

That design is why the evidence here is better than in most of this corpus: survey methodologists run split-ballot experiments on national probability samples, at scale, as a matter of routine.

Verdict

A repeat found the effect again. The other 163.

The field
54 of the 192 entries in belief and probability share this verdict.
Does it replicate?

Has Response bias been retested?

Replicated A preregistered replication found acquiescence bias again on both panels it used, but the estimate on the higher-quality panel was about a third the size.

In the original study14.70 percentage points, positive-keyed minus negative-keyed endorsement, linear probability model on the Allcott and Gentzkow items
Pooled across the replication4.90 percentage points, positive-keyed minus negative-keyed endorsement, linear probability model on the Allcott and Gentzkow items
no effect Original Original: 14.70 percentage points, positive-keyed minus negative-keyed endorsement, linear probability model on the Allcott and Gentzkow items, no interval reported Replication Replication: 4.90 percentage points, positive-keyed minus negative-keyed endorsement, linear probability model on the Allcott and Gentzkow items, no interval reported
The same scale for both, so the distance between the two points is the change the repeat found. Hover a point for its interval.

Hill and Roberts wrote a logically equivalent alternative for each of 53 questions, flipping which answer an acquiescent respondent would give. Fifty-two of the 53 points fell on the same side of the line, which they calculate has a chance probability on the order of 1 in 10 to the sixteenth. For the 35 questions whose negation was unambiguous, needing only 'not' and a verb tense, the mean gap was 8.9 percentage points; for the looser wordings it was 7.3.

The one reversal ran 12.5 points the other way, on an item about government planning of the 9/11 attacks in the China sample, which the authors attribute to their own phrasing of the negation. Cruz, Bouyamourn and Ornstein then reanalysed the study and repeated it. Their reanalysis reproduced the main effect, which they state in as many words.

Their two preregistered replications, fielded in August 2025, found it again. The positive-keying coefficient was 0.117 on the original Lucid platform, now Cint, with 2,193 respondents, and 0.049 on the Bovitz-Forthright panel with 3,036, against 0.147 in their reanalysis of the 2020 wave. All three are significant at the 1% level. The gap between panels is the finding.

Acquiescence is real on both, and three times larger where response quality is worse. One advertised correlate did not survive at all. Hill and Roberts had reported bias falling with education. Fourteen respondents whose education the vendor coded as minus 3,105 had driven that, and the authors issued a corrigendum reversing the sign. The reversed version then held on Lucid and vanished on Bovitz, which is why Cruz and colleagues read it as an artefact of inattentive and fraudulent respondents rather than a fact about educated people.

Read off Cruz, A., Bouyamourn, A., & Ornstein, J. T. (2026). Survey quality and acquiescence bias: A cautionary tale. Political Analysis, 34, 471-478.

Examples

What are some examples of Response bias?

The respondent who agrees with both halvesillustration

A questionnaire asks early on whether the service is well run and later whether it is badly run, and the same person agrees with each. The two answers describe the form of the question rather than the service.

The scale used at its endsillustration

Two people hold the same view of a course, one marking every item at the extremes and the other never leaving the middle. The gap between their totals is a habit of scale use and not a difference of opinion.

Page nine of a long formillustration

A respondent well into a survey takes the first acceptable answer to each remaining item, or the middle box. What the instrument records then depends on where a question sat in the form.

Cases marked as illustrations describe nobody in particular. They are there to make the pattern recognisable, not to report that a particular thing happened.

The experiments

What experiments is Response bias based on?

The design

The cleanest test asks the same question twice, once with the agreeable answer affirmative and once with it negative. Hill and Roberts did this for 53 questions about conspiracies and political facts across six surveys in the United States and China, drawing samples from Lucid, Qualtrics, NORC and Mechanical Turk, including a nationally representative Lucid sample of 2,055 Americans in December 2020.

On 52 of the 53 the affirmative version drew more endorsement. Asked whether the FBI director had alerted Congress about new emails on Hillary Clinton's server, 64% agreed with the affirmative wording and 22% with the negated one. Whether candidates should accept election outcomes drew 80% agreement in one direction and 52% disagreement in the other. Question format changes measurement quality as well as marginals.

Saris, Revilla, Krosnick and Shaeffer ran split-ballot multitrait-multimethod experiments inside rounds 2 and 3 of the European Social Survey, with subsamples generally above 500 cases per country. In the second round 3 experiment, on the consequences of immigration, across 23 countries, the third item scored a quality of 0.76 on an 11-point item-specific scale against 0.32 on an 11-point agree-disagree scale.

The authors decompose that gap of 0.44 themselves: reliability differed by 0.05 and validity by 0.54. Response order behaves the same way. Holbrook, Krosnick, Moore and Tourangeau meta-analysed 548 order experiments embedded in 149 Gallup telephone surveys run between 1995 and 2000, with experiment samples from 284 to 1,727 people. Recency won: 19.2% of experiments showed a significant recency effect and 1.8% a significant primacy effect, with a mean shift of 2.19 percentage points towards the last option heard.

Reading the options off a card reverses the direction. Krosnick and Alwin gave 1984 General Social Survey respondents a show card listing 13 qualities a child might have, 905 in the standard order and 446 in the reverse. 'Honest' was picked by 65.7% in the standard order against 48.4% in the reverse, a gap of 17.3 points, and 'good manners' by 26.4% against 10.1%, a gap of 16.3.

Not every item moved that way. 'Obeys his parents' ran 6.1 points the other way. How many respondents behave this way is harder to pin down. Krosnick, Villar and MacInnis, reviewing the field, put the share who answer affirmatively whatever the statement says at 10% to 20%, and quote a 1974 Schuman and Presser split ballot in which 47% agreed that most men are better suited emotionally for politics, against 33% who picked men when all three positions were named.

Both figures are quoted here at one remove from a working paper. Rates also differ by country. Harzing gave 1,581 final-year business students in 26 countries the same 69 five-point items in class between March 2001 and April 2003, and counted the share of answers at 4 or 5. That share ran from 0.39 in Japan to 0.61 in Malaysia and Taiwan, against an overall average of 0.52.

The acquiescence balance, agreement minus disagreement, ran from 0.07 in Japan to 0.44 in India and Malaysia. Those figures come from the author's accepted manuscript on her own site, not from the journal version.

10 primary research2 replication and reanalysis3 commentary and critique

Origin

Who first described Response bias, and when?

Year1991
WhoJon A. Krosnick
WhereApplied Cognitive Psychology

His paper is "Response strategies for coping with the cognitive demands of attitude measures in surveys", pages 213 to 236. It was read in full from the scanned copy on Krosnick's own Stanford page. Weak satisficing, he writes, still runs all four steps of answering, but each one less diligently and comprehensively than optimising does. Strong satisficing omits the retrieval and judgement steps altogether.

He gives three accounts of acquiescence, a personality trait, deference to a higher-status interviewer, and weak satisficing, and argues for the third. A weak satisficer asked to agree or disagree thinks only of reasons the statement might be true. Most assertions offered in surveys are reasonable, so the rule ends in agreement most of the time. Acquiescence was documented long before 1991, and Saris, Revilla, Krosnick and Shaeffer count more than a hundred studies of it. What 1991 added was one mechanism covering several response styles at once.

The claim, and the replication this entry reads its figures from.

Where it runs out

When does Response bias not apply?

Response styles being real does not make cross-national comparisons wrong. He and Van de Vijver corrected for a general response style in the 2013 Teaching and Learning International Survey, covering 76,887 teachers in 18 countries, and report that effect sizes before and after correction were very similar for all constructs and that correlations before and after were mostly higher than 0.90.

The score rank order of countries, they write, is not affected by the corrections. Whether item-specific formats beat agree-disagree ones is disputed, and this is the sharpest disagreement on the page. Lelkes and Weiss compared four political efficacy items in both formats across the two waves of the 2012 American National Election Studies, which ran 5,916 pre-election and 5,513 post-election interviews.

Test-retest polychoric correlations averaged 0.63 for the agree-disagree items and 0.63 for the construct-specific ones. On criterion validity the average difference between the formats was minus 0.02 for political knowledge, minus 0.05 for stated chance of voting and minus 0.03 for political activity, each favouring the construct-specific form and none of them large. Construct-specific questions were no more valid among respondents low in verbal ability, high in agreeableness or interviewed face to face, which are the three groups the acquiescence accounts single out.

The authors do not conclude that acquiescence is harmless. They say they are sceptical of that reading, and offer two others: construct-specific wording may not be the right remedy, or this particular battery may be a poor test of it. Liu, Lee and Conrad, read only as the abstract Oxford shows above its purchase prompt, report that extreme responding exists in both agree-disagree and item-specific scales and behaves as a single style across the two.

Two more caveats. The European Social Survey quality estimates come from a model that fixes one item-specific method's validity at exactly 1 in each experiment, a property of the identification rather than a finding, and the printed standard deviation of 0 on those cells says so. And Harzing's 26-country figures are from an accepted manuscript dated May 2006, not from the article as the journal printed it, so the published numbers may differ.

Access, as tested on 21 September 2026. Lelkes and Weiss is gold open access, and the publisher's PDF returns a Cloudflare managed bot challenge headed 'Just a moment...', not a paywall; an earlier note here called it a 403 and stopped there. The article was read in full from the University of Amsterdam repository, whose cover page marks it the final published version under CC BY-NC.

Krosnick's 1991 paper is closed at Wiley, which also answers a bot challenge, but the scan on his Stanford page is open and was read. Harzing's paper is closed at Sage, and her own site serves the accepted manuscript. The Springer PDF of Clifford and Engelhardt is open access behind a JavaScript client challenge that a plain request passes.

Oxford serves He and Van de Vijver's open-access article behind the same kind of challenge, which no command-line request here got through, so its figures were read off the publisher's own article page through a fetching tool and nothing beyond the quoted sentences is claimed from it. The deposits for that paper at pedocs and at North-West University hold no file, only a link back to the DOI.

Liu, Lee and Conrad is a genuine paywall, priced per article. Searched the same day at Crossref, OpenAlex, Unpaywall and the open web, no direct repeat of Krosnick and Alwin's 1984 show-card experiment was found.

Commonly misread as

How is Response bias commonly misread?

The first error is filing this with the judgement biases. Nothing here says people reason badly. It says a rating scale is a device with its own behaviour, and that the same person gives a different answer to 'agree or disagree' than to a question that names the options. Entry 443 is the same point made with one pair of verbs, and entry 467 makes it with the numbers printed beside the boxes.

The second is the belief that reverse-coding half the items cancels acquiescence out. Clifford and Engelhardt ran 160 acquiescence tests across four studies and seven constructs, and found that balanced scales do not remove the bias. Modelling acquiescence explicitly changed the size, and sometimes the direction, of the relationship between constructs. Their summary puts the average difference between correlations of same-direction and opposite-direction scales at 0.26, reaching 0.35 in the study fielded through a commercial panel, while the body of the paper gives 0.19 across all studies and constructs.

Of the 160 pairs, 31% differ in sign. The third is treating acquiescence as purely a personality trait. Weijters, Geuens and Schillewaert invited an internet panel to answer two non-overlapping sets of items a year apart. The first wave returned 1,506 usable cases and the second only 604, and the model was fitted over all of them by full-information maximum likelihood.

Between 55% and 65% of the variance in the wave-specific response style factors was time-invariant, so a stable component exists. Demographics explained 4.2% of the stable acquiescence factor and 8.3% of the extreme responding factor, and Holbrook and colleagues found question comprehension difficulty predicted order effects strongly, so the situation carries much of the weight. Those figures come from the Ghent University working paper version rather than the Psychological Methods article.

The fourth is reading extreme responding as a fixed cultural signature. Cerda and Basar found Hispanic respondents used both endpoints on four or five of five items 30% of the time against 18% for non-Hispanics on a 10-point scale, in cells of 146 each, and no such gap on a 5-point scale. Format and style interact, which is why the remedy is usually a better question rather than a statistical correction.

Sources

Sources for Response bias

  1. 1Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236. Read in full from the scanned copy on Krosnick's Stanford page at the URL given; Wiley answers a bot challenge rather than serving the article.primary
  2. 2Hill, S. J., & Roberts, M. E. (2023). Acquiescence bias inflates estimates of conspiratorial beliefs and political misperceptions. Political Analysis, 31(4), 575-590.primary
  3. 3Hill, S. J., & Roberts, M. E. (2025). Acquiescence bias inflates estimates of conspiratorial beliefs and political misperceptions - CORRIGENDUM. Political Analysis, 33(2), 178-180. The authors' own correction: 14 respondents whose education the vendor coded as -3105 reversed the sign of the education interaction, and they ask readers to revise their summary accordingly.primary
  4. 4Cruz, A., Bouyamourn, A., & Ornstein, J. T. (2026). Survey quality and acquiescence bias: A cautionary tale. Political Analysis, 34, 471-478. Preregistered as AsPredicted no. 239215.replication
  5. 5Saris, W., Revilla, M., Krosnick, J. A., & Shaeffer, E. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61-79.primary
  6. 6Holbrook, A. L., Krosnick, J. A., Moore, D., & Tourangeau, R. (2007). Response order effects in dichotomous categorical questions presented orally: The impact of question and respondent attributes. Public Opinion Quarterly, 71(3), 325-348.meta-analysis
  7. 7Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201-219. Read as the scan hosted as GSS Methodological Report 45, which carries the journal's pagination.primary
  8. 8Clifford, S., & Engelhardt, A. M. (2026). Acquiescence bias and criterion validity: Problems and potential solutions for agree-disagree scales. Political Behavior. Open access at Springer, behind a client challenge that a plain request passes.primary
  9. 9Weijters, B., Geuens, M., & Schillewaert, N. (2010). The stability of individual response styles. Psychological Methods, 15(1), 96-110. Figures read from the Ghent University working paper 08/547, not the journal version.primary
  10. 10He, J., & Van de Vijver, F. J. R. (2015). Effects of a general response style on cross-cultural comparisons: Evidence from the Teaching and Learning International Survey. Public Opinion Quarterly, 79(S1), 267-290. Open access, but Oxford answers a bot challenge to command-line requests, so the sentences quoted here were read off the publisher's article page through a fetching tool. The pedocs and North-West University records hold no file.critique
  11. 11Lelkes, Y., & Weiss, R. (2015). Much ado about acquiescence: The relative validity and reliability of construct-specific and agree-disagree questions. Research & Politics, 2(3). Read in full from the University of Amsterdam repository copy at the URL given, marked there as the final published version under CC BY-NC. The publisher's PDF returns a Cloudflare bot challenge, not a paywall.critique
  12. 12Harzing, A.-W. (2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6(2), 243-266. Read as the author's accepted manuscript, dated May 2006, on her own site. The journal version is closed at Sage and was not obtained.primary
  13. 13Cerda, M., & Basar, I. (2010). Extreme response style among U.S. Hispanics vs. non-Hispanics. Survey Practice, 3(4), 1-8.primary
  14. 14Liu, M., Lee, S., & Conrad, F. G. (2015). Comparing extreme response styles between agree-disagree and item-specific scales. Public Opinion Quarterly, 79(4), 952-975. Abstract only: Oxford shows the abstract above a per-article purchase prompt, and no repository copy is recorded.primary
  15. 15Krosnick, J. A., Villar, A., & MacInnis, B. (2011). Do introductory sentences cause acquiescence response bias in survey questions? Stanford University working paper, dated August 2011. Source for the 10% to 20% figure and the Schuman and Presser split ballot, both quoted there at one remove. The Stanford link now serves a Google Drive viewer, and the file itself was downloaded from Drive.secondary

Every claim on this page was held against these sources on 2026-09-21. Nothing here is written from memory. Written and checked by Krishna Chagti; how entries are written and corrected. Found a mistake? Report it, with the sentence and a source.

About Response bias

Is Response bias real?

A preregistered replication found acquiescence bias again on both panels it used, but the estimate on the higher-quality panel was about a third the size. The effect sizes from the original study and from the replication are printed side by side above, so the comparison is visible rather than asserted.