Choosing the Number of Categories in Agree–Disagree Scales

Although agree–disagree (AD) rating scales suffer from acquiescence response bias, entail enhanced cognitive burden, and yield data of lower quality, these scales remain popular with researchers due to practical considerations (e.g., ease of item preparation, speed of administration, and reduced administration costs). This article shows that if researchers want to use AD scales, they should offer 5 answer categories rather than 7 or 11, because the latter yield data of lower quality. This is shown using data from four multitrait-multimethod experiments implemented in the third round of the European Social Survey. The quality of items with different rating scale lengths were computed and compared.

[1]  J. Dawes.  Do Data Characteristics Change According to the Number of Scale Points Used? An Experiment Using 5-Point, 7-Point and 10-Point Scales , 2008 .

[2]  Marcel Adam Just,et al.  Sentence comprehension: A psycholinguistic processing model of verification. , 1975 .

[3]  R. P. McDonald,et al.  Evaluation of Measurement Instruments by Meta-Analysis of Multitrait Multimethod Studies. , 1992 .

[4]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[5]  Willem E. Saris,et al.  Design, Evaluation, and Analysis of Questionnaires for Survey Research: Saris/Design , 2007 .

[6]  Willem E. Saris,et al.  Testing Structural Equation Models or Detection of Misspecifications? , 2009 .

[7]  J. Fodor.  Psychology and Language. , 1970 .

[8]  Robert L. Linn,et al.  Path analysis: Psychological examples. , 1970 .

[9]  W. R. Garner,et al.  Rating scales, discriminability, and information transmission. , 1960, Psychological review.

[10]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[11]  J. Krosnick,et al.  Comparing questions with agree/ disagree response options to questions with item-specific response options , 2010 .

[12]  K. Jöreskog.  Statistical analysis of sets of congeneric tests , 1971 .

[13]  Christopher Winship,et al.  REGRESSION MODELS WITH ORDINAL VARIABLES , 1984 .

[14]  Duane F. Alwin,et al.  INFORMATION TRANSMISSION IN THE SURVEY INTERVIEW: NUMBER OF RESPONSE CATEGORIES AND THE RELIABILITY OF ATTITUDE MEASUREMENT , 1992 .

[15]  D. Alwin,et al.  Feeling Thermometers Versus 7-Point Scales , 1997 .

[16]  A. Tellegen,et al.  PERSONALITY PROCESSES AND INDIVIDUAL DIFFERENCES An Alternative "Description of Personality": The Big-Five Factor Structure , 2022 .

[17]  Jon A. Krosnick,et al.  Designing Rating Scales for Effective Measurement in Surveys , 1997 .

[18]  R. Cudeck.  Msultiplicative Models and MTMM Matrices , 1988 .

[19]  Willem E. Saris,et al.  Memory effects in MTMM studies , 1995 .

[20]  Alex C. Michalos,et al.  Andrews, Frank M. , 2014 .

[21]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[22]  K. Jöreskog.  A general method for analysis of covariance structures , 1970 .

[23]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[24]  Willem E. Saris,et al.  Different Explanations for Correlated Disturbance Terms in MTMM Studies , 2003 .

[25]  F. Al-Shamali,et al.  Author Biographies. , 2015, Journal of social work in disability & rehabilitation.

[26]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[27]  Vanessa Hertzog,et al.  Design Evaluation And Analysis Of Questionnaires For Survey Research , 2016 .

[28]  Jon A. Krosnick,et al.  The Reliability of Survey Attitude Measurement , 1991 .

[29]  Duane F. Alwin,et al.  Evaluating the Reliability and Validity of Survey Interview Data Using the MTMM Approach , 2011 .

[30]  Annette Scherpenzeel,et al.  The Validity and Reliability of Survey Questions , 1997 .

[31]  Annette Scherpenzeel,et al.  Validity and Reliability of Subjective Social Indicators: The effect of different measures of association , 1998 .

[32]  G. Loosveldt,et al.  Measuring meaningful data in social research , 2007 .

[33]  John C. Leggett,et al.  Caste, Class, and Deference in the Research Interview , 1960, American Journal of Sociology.

[34]  Jon A. Krosnick,et al.  Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format , 1993 .

[35]  Eric R. Ziegel,et al.  Improving Survey Questions , 1997 .

[36]  Willem E. Saris,et al.  Evaluation of Measurement Instruments Using a Structural Modeling Approach , 2011 .

[37]  Willem E. Saris,et al.  Fit of Different Models for Multitrait-Multimethod Experiments , 2002 .

[38]  T. Trabasso,et al.  Storage and verification stages in processing concepts. , 1971 .

[39]  Karl G. Jöreskog,et al.  LISREL 7: A guide to the program and applications , 1988 .

[40]  Floyd J. Fowler,et al.  Improving Survey Questions: Design and Evaluation , 1995 .

[41]  M. Browne,et al.  The decomposition of multitrait-multimethod matrices. , 1984, The British journal of mathematical and statistical psychology.

[42]  Willem E. Saris,et al.  8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design , 2004 .

[43]  W. Saris,et al.  Categorization and measurement quality. The choice between Pearson and Polychoric correlations , 1995 .

[44]  F. M. Andrews.  Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach , 1984 .

[45]  Studies of the polychoric correlation and other correlation measures for ordinal variables , 1992 .

[46]  E. Davidov,et al.  Testing the Stability of an Acquiescence Style Factor Behind Two Interrelated Substantive Variables in a Panel Design , 2008 .

[47]  A. C. Scherpenzeel.  A Question of Quality. Evaluating survey questions by multitrait-multimethod studies , 1995 .

[48]  P. Marsden.  Margins of Error: A Study of Reliability in Survey Measurement , 2011 .

[49]  D. Alwin.  Approaches to the Interpretation of Relationships in the Multitrait-Multimethod Matrix , 1973 .