An experiment testing six formats of 101-point rating scales

Although 101-point feeling thermometer questions are frequently used in population surveys and political polls, the measurement error associated with this question type has raised some concerns. One challenge of using this type of question arises from the tendency for respondents to round their answers. That is, rather than providing a precise answer, respondents provide an answer that is divisible by 5 which on at least some occasions is rounded up or down from a more precise (unrounded) value. Several national surveys currently use an open-ended numeric text input format for the feeling thermometer questions, which may exacerbate the response rounding. This study presents findings from two Web survey experiments that examine six formats of 101-point rating scales. The findings show that visual analog scales (VAS) result in fewer rounded answers than open-ended numeric text input, while item nonresponse rates are similar across these questions types. Also, responding to VAS tends to be easier than open-ended format. This study concludes by discussing the application of VAS for feeling thermometer questions and future research directions. Compared six types of 101-point feeling thermometer questions.Visual analog scales elicit fewer rounded answers than open-ended questions.Completion time is faster for visual analog scales and open-ended questions.Item nonresponse rates are similar across question types.

[1]  Mark P. Jensen,et al.  Validity of four pain intensity rating scales , 2011, PAIN®.

[2]  Yichen Wang,et al.  Data collection mode effect on feeling thermometer questions: A comparison of face-to-face and Web surveys , 2015, Comput. Hum. Behav..

[3]  Augusto Caraceni,et al.  Studies comparing Numerical Rating Scales, Verbal Rating Scales, and Visual Analogue Scales for assessment of pain intensity in adults: a systematic literature review. , 2011, Journal of pain and symptom management.

[4]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[5]  Michael D. Buhrmester,et al.  Amazon's Mechanical Turk , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[6]  Roger Tourangeau,et al.  Evaluating the Effectiveness of Visual Analog Scales , 2006 .

[7]  L. Hedges,et al.  Reports of elapsed time: bounding and rounding processes in estimation. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[8]  Gilbert J Fanciullo,et al.  Computerized dynamic assessment of pain: comparison of chronic pain patients and healthy controls. , 2004, Pain medicine.

[9]  F. Conrad,et al.  Strategies for estimating behavioural frequency in survey interviews. , 1998, Memory.

[10]  Ulf-Dietrich Reips,et al.  Sliders for the Smart: Type of Rating Scale on the Web Interacts With Educational Level , 2011 .

[11]  Roger Tourangeau,et al.  Evaluating the Effectiveness of Visual Analog Scales : A Web Experiment , 2006 .

[12]  P. Bijur,et al.  Prospective validation of clinically important changes in pain severity measured on a visual analog scale. , 2001, Annals of emergency medicine.

[13]  A. Calin,et al.  Assessment of disease severity (in terms of function) using the internet. , 2004, The Journal of rheumatology.

[14]  P. Bijur,et al.  Reliability of the visual analog scale for measurement of acute pain. , 2001, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[15]  M. Jensen,et al.  Interpretation of visual analog scale ratings and change scores: a reanalysis of two clinical trials of postoperative pain. , 2003, The journal of pain : official journal of the American Pain Society.

[16]  Bruce Thompson,et al.  Score Reliability in Webor Internet-Based Surveys: Unnumbered Graphic Rating Scales versus Likert-Type Scales , 2001 .

[17]  C. Wells,et al.  Comparison of three rating scales for measuring subjective phenomena in clinical research. I. Use of experimentally controlled auditory stimuli. , 2004, Archives of medical research.

[18]  Willem E. Saris,et al.  8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design , 2004 .

[19]  C. Lumsden,et al.  Portable mood mapping: the validity and reliability of analog scale displays for mood assessment via hand-held computer , 2003, Psychiatry Research.

[20]  J. Krosnick Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[21]  Siddharth Suri,et al.  Conducting behavioral research on Amazon’s Mechanical Turk , 2010, Behavior research methods.

[22]  L. Rips,et al.  The Psychology of Survey Response , 2000 .