How constrained a response: a comparison of binary, ordinal and metric answer formats

Abstract A question is the main measurement instrument in the social sciences. Yet no conclusive results exist with respect to the suitability of alternative answer formats for typical constructs studied in this field. Furthermore, no prior studies have used real answers from respondents to investigate differences in responses as a consequence of answer formats, typically assuming the way in which individuals translate their responses from one answer format to another. In this study we make a first step towards filling the above two gaps. We investigate answer format effects for two different constructs (attitudes, intentions) and three formats (binary, ordinal, metric) using a repeat measurement design. Results indicate that formats lead to the same managerial interpretations with the same reliability; differ in speed and perceived speed while being perceived as equally simple, pleasant, and useful to express feelings. Binary and metric answer formats are found to represent interesting alternatives to the predominantly used ordinal format, especially when speed of completion or the availability of metric data for analyses is essential.

[1]  D. Dillman.  Mail and telephone surveys : the total design method , 1979 .

[2]  Seymour Sudman,et al.  Improving Interview Method and Questionnaire Design , 1979 .

[3]  T. R. F. Oaster,et al.  Number of Alternatives per Choice Point and Stability of Likert-Type Scales , 1989 .

[4]  D. Cicchetti,et al.  Comparative reliability of categorical and analogue rating scales in the assessment of psychiatric symptomatology , 1979, Psychological Medicine.

[5]  R. Dunlap,et al.  The “New Environmental Paradigm” , 1978 .

[6]  An Investigation of the Effect of the Number of Scale Intervals on Principal Components Factor Analysis , 1974 .

[7]  Lei Chang.  A Psychometric Evaluation of 4-Point and 6-Point Likert-Type Scales in Relation to Reliability and Validity , 1994 .

[8]  S. S. Komorita,et al.  Number of Scale Points and the Reliability of Scales , 1965 .

[9]  G. Hancock,et al.  The effect of scale manipulations on validity: targetting frequency rating scales for anticipated performance levels. , 1991, Applied ergonomics.

[10]  A. Colman,et al.  Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. , 2000, Acta psychologica.

[11]  R. Dunlap,et al.  Measuring Endorsement of the New Ecological Paradigm: A Revised NEP Scale , 2000 .

[12]  M. Wedel,et al.  Market Segmentation: Conceptual and Methodological Foundations , 1997 .

[13]  R. Dunlap,et al.  THE NEW ENVIRONMENTAL PARADIGM: 5 PROPOSED MEASURING INSTRUMENT AND PRELIMINARY RESULTS , 1978 .

[14]  J. Jacoby,et al.  Three-Point Likert Scales Are Good Enough , 1971 .

[15]  Donald R. Lehmann,et al.  Are Three-Point Scales Always Good Enough? , 1972 .

[16]  Karen Cortell Reisman.  "The art of asking questions". , 2008, Texas dental journal.

[17]  P. McCullagh.  Regression Models for Ordinal Data , 1980 .

[18]  D. Bates,et al.  Mixed-Effects Models in S and S-PLUS , 2001 .

[19]  Bettina Grün,et al.  Time Efficient Brand Image Measurement - Is Binary Format Sufficient to Gain the Market Insight Required? , 2004 .

[20]  Sara Dolnicar,et al.  Simplifying Three-way Questionnaires - Do the Advantages of Binary Answer Categories Compensate for the Loss of Information? , 2003 .

[21]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[22]  W. Martin.  Effects of Scaling on the Correlation Coefficient: Additional Considerations , 1978 .

[23]  H. Harman.  Modern factor analysis , 1961 .

[24]  Martin E. P. Seligman,et al.  The attributional Style Questionnaire , 1982, Cognitive Therapy and Research.

[25]  M. Seligman,et al.  The Attributional Style Questionnaire1 , 1982 .

[26]  Paul E. Green,et al.  Rating Scales and Information Recovery—How Many Scales and Response Categories to Use? , 1970 .

[27]  A. W. Bendig.  Reliability and the number of rating-scale categories. , 1954 .

[28]  D. Peabody,et al.  Two components in bipolar scales: direction and extremeness. , 1962, Psychological review.

[29]  R. H. Finn.  Effects of Some Variations in Rating Scale Characteristics on the Means and Reliabilities of Ratings , 1972 .

[30]  P. Symonds,et al.  On the Loss of Reliability in Ratings Due to Coarseness of the Scale. , 1924 .

[31]  J. Jacoby,et al.  Is There an Optimal Number of Alternatives for Likert Scale Items? Study I: Reliability and Validity , 1971 .

[32]  R. Cattell.  The Scree Test For The Number Of Factors. , 1966, Multivariate behavioral research.

[33]  G. Katona,et al.  The Art of Asking Questions , 1951 .

[34]  Adam Finn,et al.  Reliability Assessment and Optimization of Marketing Measurement , 1997 .

[35]  James O. Ramsay,et al.  The effect of number of categories in rating scales on precision of estimation of scale values , 1973 .

[36]  Larry Percy,et al.  An Argument in Support of Ordinary Factor Analysis of Dichotomous Variables , 1976 .

[37]  Eli P. Cox,et al.  The optimal number of response alternatives for a scale: A review. , 1980 .

[38]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[39]  Jarl K. Kampen,et al.  The Ordinal Controversy Revisited , 2000 .

[40]  Daniel J. Mundfrom,et al.  Minimum Sample Size Recommendations for Conducting Factor Analyses , 2005 .

[41]  Cees van der Eijk,et al.  Measuring agreement in ordered rating scales. , 2001 .

[42]  Eli P. Cox,et al.  The Optimal Number of Response Alternatives for a Scale: A Review , 1980 .

[43]  S. Komorita,et al.  ATTITUDE CONTENT, INTENSITY, AND THE NEUTRAL POINT ON A LIKERT SCALE. , 1963, The Journal of social psychology.

[44]  Seymour Sudman,et al.  Improving Interview Method and Questionnaire Design. , 1982 .

[45]  L. Cronbach.  Further Evidence on Response Sets and Test Design , 1950 .

[46]  Jacob Cohen.  The Cost of Dichotomization , 1983 .

[47]  W. Martin.  The Effects of Scaling on the Correlation Coefficient: A Test of Validity , 1973 .