Some Methodological Uses of Responses to Open Questions and Other Verbatim Comments in Quantitative Surveys

The use of open-ended questions in survey research has a very long history. In this paper, building on the work of Paul F. Lazarsfeld and Howard Schuman, we review the methodological uses of open-ended questions and verbatim responses in surveys. We draw on prior research, our own and that of others, to argue for increasing the use of open-ended questions in quantitative surveys. The addition of open-ended questions – and the capture and analysis of respondents’ verbatim responses to other types of questions – may yield important insights, not only into respondents’ substantive answers, but also into how they understand the questions we ask and arrive at an answer. Adding a limited number of such questions to computerized surveys, whether self- or interviewer-administered, is neither expensive nor time-consuming, and in our experience respondents are quite willing and able to answer such questions.

[1]  R. Porst,et al.  Wären Sie vielleicht bereit, sich gegebenenfalls noch einmal befragen zu lassen? Oder: Gründe für die Teilnahme an Panelbefragungen , 1995 .

[2]  F. Conrad,et al.  Examples in Open-Ended Survey Questions , 2016 .

[3]  Mick P Couper,et al.  Risk of Disclosure, Perceptions of Risk, and Concerns about Privacy and Confidentiality as Factors in Survey Participation. , 2008, Journal of official statistics.

[4]  M. Couper,et al.  Linking Survey and Administrative Records , 2012, Sociological methods & research.

[5]  Charles F. Cannell,et al.  Using behavioral coding to identify cognitive problems with survey questions. , 1996 .

[6]  Jennifer Hess,et al.  THE ROLE OF RESPONDENT DEBRIEFING QUESTIONS IN QUESTIONNAIRE DEVELOPMENT , 1998 .

[7]  M. Couper,et al.  Informed Consent for Web Paradata Use. , 2013, Survey research methods.

[8]  Marek Fuchs,et al.  Asking for Numbers and Quantities: Visual Design Effects in Paper&Pencil Surveys , 2009 .

[9]  Theresa J. DeMaio,et al.  Cognitive interviewing techniques: In the lab and in the field. , 1996 .

[10]  N. Bradburn,et al.  How to Ask Questions about Drinking and Sex: Response Effects in Measuring Consumer Behavior , 1976 .

[11]  E. Singer,et al.  Paying Respondents for Survey Participation , 2001 .

[12]  Mick P Couper,et al.  Experimental Studies of Disclosure Risk, Disclosure Harm, Topic Sensitivity, and Survey Participation. , 2010, Journal of official statistics.

[13]  Sridevi Sattaluri,et al.  Computer Audio-Recorded Interviewing (CARI): A Tool for Monitoring Field Interviewers and Improving Field Data Collection , 2008 .

[14]  W. Belson,et al.  The design and understanding of survey questions , 1982 .

[15]  M. Rita Thissen,et al.  Computer Audio-Recorded Interviewing as a Tool for Survey Research , 2014 .

[16]  M. Couper Survey introductions and data quality , 1997 .

[17]  Brady T. West,et al.  Factors Affecting the Accuracy of Interviewer Observations Evidence from the National Survey of Family Growth , 2013 .

[18]  M. Rita Thissen,et al.  Computer Audio Recording: A Practical Technology for Improving Survey Quality , 2013 .

[19]  E. Singer,et al.  Exploring the Meaning of Consent: Participation in Research and Beliefs about Risks and Benefits , 2003 .

[20]  Charles C. Brown,et al.  The Effects of Respondents’ Consent to Be Recorded on Interview Length and Data Quality in a National Panel Study , 2015, Field methods.

[21]  M. Galesic,et al.  Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey , 2009 .

[22]  Patrick Sturgis,et al.  The Scope for Reducing Refusals in Household Surveys: An Investigation Based on Transcripts of tape-recorded Doorstep Interactions , 1998 .

[23]  H. Schuman,et al.  The random probe: a technique for evaluating the validity of closed questions. , 1966, American sociological review.

[24]  S. Presser,et al.  Question Wording as an Independent Variable in Survey Analysis , 1977 .

[25]  John G. Geer,et al.  DO OPEN-ENDED QUESTIONS MEASURE “SALIENT” ISSUES? , 1991 .

[26]  Matthias Schonlau,et al.  Semi-automated categorization of open-ended questions , 2016 .

[27]  Brady T. West,et al.  An examination of the quality and utility of interviewer observations in the National Survey of Family Growth , 2013 .

[28]  Ewan Klein,et al.  Natural Language Processing with Python , 2009 .

[29]  Frederick G. Conrad,et al.  Effects of speech rate, pitch, and pausing on survey participation decisions , 2011 .

[30]  Roger Tourangeau,et al.  Designing Input Fields for Non-Narrative Open-Ended Responses in Web Surveys. , 2011, Journal of official statistics.

[31]  Howard Schuman Method and Meaning in Polls and Surveys , 2008 .

[32]  Stanley Presser,et al.  The Open and Closed Question , 1979 .

[33]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[34]  Jolene D. Smyth,et al.  Open-Ended Questions in Web Surveys Can Increasing the Size of Answer Boxes and Providing Extra Verbal Instructions Improve Response Quality? , 2009 .

[35]  P. Lazarsfeld THE CONTROVERSY OVER DETAILED INTERVIEWS—AN OFFER FOR NEGOTIATION , 1944 .

[36]  Stefano M. Iacus Big Data and Social Science - A Practical Guide to Methods and Tools , 2017 .

[37]  Lars Kaczmirek,et al.  Asking Probing Questions in Web Surveys , 2012 .

[38]  Mick P. Couper,et al.  Technology and the Survey Interview/Questionnaire , 2007 .

[39]  Brad Edwards,et al.  Using Cari Tools To Understand Measurement Error , 2010 .

[40]  M. Pfeifer Standardization And Tacit Knowledge Interaction And Practice In The Survey Interview , 2016 .

[41]  Mick P. Couper,et al.  Race in the Live and the Virtual Interview: Racial Deference, Social Desirability, and Activation Effects in Attitude Surveys , 2003 .

[42]  David L. Vannette,et al.  Interviewer speech and the success of survey invitations , 2013 .

[43]  Brady T. West,et al.  A Practical Technique for Improving the Accuracy of Interviewer Observations of Respondent Characteristics , 2015 .

[44]  James M. Dahlhamer,et al.  Privacy concerns, too busy, or just not interested: using doorstep concerns to predict survey nonresponse , 2008 .

[45]  Matt E. Jans,et al.  Trends in Income nonresponse over two decades , 2010 .

[46]  Susan Purdon,et al.  Can you hear me knocking? and investigation into the impact of interviewers on survey response rates , 1997 .

[47]  Lucilla Tan An Introduction to the Contact History Instrument (CHI) for the Consumer Expenditure Survey , 2011 .

[48]  M. Couper,et al.  Characteristics of Physical Measurement Consent in a Population-Based Survey of Older Adults , 2010, Medical care.

[49]  U. Census,et al.  Evaluating the Computer Audio-Recorded Interviewing ( CARI ) Household Wellness Study ( HWS ) Field Test , 2007 .

[50]  Lars Kaczmirek,et al.  Testing the Validity of Gender Ideology Items by Implementing Probing Questions in Web Surveys , 2013 .

[51]  N. C. Schaeffer,et al.  Requests, Blocking Moves, and Rational (Inter)action in Survey Introductions. , 2010, American sociological review.

[52]  Marika Wenemark,et al.  The respondent’s perspective in health-related surveys , 2011 .

[53]  Glenn D. Israel Effects of Answer Space Size on Responses to Open-ended Questions in Mail Surveys , 2010 .

[54]  F. Conrad,et al.  The Effects of Providing Examples in Survey Questions , 2014 .

[55]  R. Tourangeau,et al.  CONTRAST EFFECTS AND SUBTRACTION IN PART-WHOLE QUESTIONS , 1994 .

[56]  E. Singer Toward a benefit-cost theory or survey participation: Evidence, further tests, and implications , 2011 .

[57]  M. Couper,et al.  Methods for Testing and Evaluating Survey Questionnaires , 2004 .

[58]  R. Groves,et al.  A Theory-Guided Interviewer Training Protocol Regarding Survey Participation , 2001 .

[59]  Alicia O'Cathain,et al.  "Any other comments?" Open questions on questionnaires – a bane or a bonus to research? , 2004, BMC medical research methodology.

[60]  Lars Kaczmirek,et al.  Cognitive Probes in Web Surveys , 2014 .

[61]  Jennifer Dykema,et al.  The Effect of Interviewer and Respondent Behavior on Data Quality: Analysis of Interaction Coding in a Validation Study , 1997 .