The Science of Asking Questions

■ Abstract Survey methodologists have drawn on and contributed to research by cognitive psychologists, conversation analysts, and others to lay a foundation for the science of asking questions. Our discussion of this work is structured around the decisions that must be made for two common types of inquiries: questions about events or behaviors and questions that ask for evaluations or attitudes. The issues we review for behaviors include definitions, reference periods, response dimensions, and response categories. The issues we review for attitudes include bipolar versus unipolar scales, number of categories, category labels, don’t know filters, and acquiescence. We also review procedures for question testing and evaluation.

[1]  N. C. Schaeffer,et al.  Conversation and Cognition: From paradigm to prototype and back again: interactive aspects of ‘cognitive processing’ in standardized survey interviews , 2005 .

[2]  N. C. Schaeffer,et al.  A Multiple‐Method Approach to Improving the Clarity of Closely Related Concepts: Distinguishing Legal and Physical Custody of Children , 2004 .

[3]  P. Converse.  The Nature of Belief Systems in Mass Publics , 2004 .

[4]  N. Schwarz,et al.  Thinking About Answers: The Application of Cognitive Processes to Survey Methodology , 1995, Quality of Life Research.

[5]  Norbert Schwarz,et al.  Making sense of standardized survey questions: The influence of reference periods and their repetition , 2002 .

[6]  J. van der Zouwen,et al.  Standardization and tacit knowledge : interaction and practice in the survey interview , 2002 .

[7]  F P Stafford,et al.  Event history calendars and question list surveys: a direct comparison of interviewing methods. , 2001, Public opinion quarterly.

[8]  Robert Cameron Mitchell,et al.  The Impact of "No Opinion" Response Options on Data Quality: Non-Attitude Reduction or an Invitation to Satisfice? , 2001 .

[9]  Jon A. Krosnick,et al.  Middle Alternatives, Acquiescence, and the Quality of Questionnaire Data , 2000 .

[10]  N. C. Schaeffer,et al.  Events, Instruments, and Reporting Errors , 2000, American Sociological Review.

[11]  Roger Tourangeau,et al.  Cognition and Survey Research , 2000, Technometrics.

[12]  Jennifer M. Talarico,et al.  Decomposition can harm the accuracy of behavioural frequency reports , 2000 .

[13]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[14]  F. Conrad,et al.  Clarifying question meaning in a household telephone survey. , 2000, Public opinion quarterly.

[15]  E. Singer,et al.  PREDICTING TEST—RETEST RELIABILITY FROM BEHAVIOR CODING , 1999 .

[16]  Robert F. Belli,et al.  Reducing vote overreporting in surveys : Social desirability, memory failure, and source monitoring , 1999 .

[17]  Roger Tourangeau,et al.  Are reporting errors due to encoding limitations or retrieval failure? Surveys of child vaccination as a case study , 1999 .

[18]  N. Schwarz,et al.  Looking back at anger: reference periods change the interpretation of emotion frequency questions. , 1998, Journal of personality and social psychology.

[19]  Susan M. Rogers,et al.  Automated Self-interviewing and the Survey Measurement of Sensitive Behaviors , 1998 .

[20]  F. Conrad,et al.  Strategies for estimating behavioural frequency in survey interviews. , 1998, Memory.

[21]  Norbert Schwarz,et al.  FORMAL FEATURES OF RATING SCALES AND THE INTERPRETATION OF QUESTION MEANING , 1998 .

[22]  Nancy A. Mathiowetz.  Respondent Expressions of Uncertainty: Data Source for Imputation , 1998 .

[23]  Bärbel Knäuper,et al.  Filter Questions and Question Interpretation: Presuppositions at Work , 1998 .

[24]  Eric R. Ziegel,et al.  Survey measurement and process quality , 1998 .

[25]  S. Presser,et al.  Data collection mode and social desirability bias in self-reported religious attendance : Church attendance in the United States , 1998 .

[26]  Jennifer Hess,et al.  THE ROLE OF RESPONDENT DEBRIEFING QUESTIONS IN QUESTIONNAIRE DEVELOPMENT , 1998 .

[27]  Jeffrey C. Moore,et al.  Questionnaire Design Effects on Interview Outcomes , 1998 .

[28]  G. Menon,et al.  Are the Parts Better than the Whole? The Effects of Decompositional Questions on Judgments of Frequent Behaviors , 1997 .

[29]  Edith D. de Leeuw,et al.  Survey Measurement and Process Quality: Lyberg/Survey , 1997 .

[30]  Jennifer Dykema,et al.  The Effect of Interviewer and Respondent Behavior on Data Quality: Analysis of Interaction Coding in a Validation Study , 1997 .

[31]  Roger Tourangeau,et al.  Effects of Interview Mode on Sensitive Questions in a Fertility Survey , 1997 .

[32]  Frederick G. Conrad,et al.  Does Conversational Interviewing Reduce Survey Measurement Error , 1997 .

[33]  D. Alwin,et al.  Feeling Thermometers Versus 7-Point Scales , 1997 .

[34]  M. Schober,et al.  Does conversational interviewing reduce survey measurement error?. Public opinion quarterly, . , 1997 .

[35]  Seymour Sudman,et al.  Thinking about Answers: The Application of Cognitive Process to Survey Methodology , 1996 .

[36]  Jeff Dominitz,et al.  Perceptions of Economic Insecurity: Evidence from the Survey of Economic Expectations , 1996 .

[37]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[38]  Charles F. Cannell,et al.  Using behavioral coding to identify cognitive problems with survey questions. , 1996 .

[39]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[40]  N. C. Schaeffer.  Errors of Experience: Response Errors in Reports about Child Support and Their Implications for Questionnaire Design , 1994 .

[41]  Stanley Presser,et al.  SURVEY PRETESTING: DO DIFFERENT METHODS PRODUCE DIFFERENT RESULTS? , 1994 .

[42]  Norbert Schwarz,et al.  Judgment in a Social Context: Biases, Shortcomings, and the Logic of Conversation , 1994 .

[43]  G. Menon,et al.  The Effects of Accessibility of Information in Memory on Judgments of Behavioral Frequencies , 1993 .

[44]  Jon A. Krosnick,et al.  Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format , 1993 .

[45]  D. Alwin,et al.  No-Opinion Filters and Attitude Measurement Reliability , 1993 .

[46]  Judith M. Tanur,et al.  Questions About Questions: Inquiries into the Cognitive Bases of Surveys , 1993 .

[47]  F. J. Fowler,et al.  How unclear terms affect survey data. , 1992, Public opinion quarterly.

[48]  M. Traugott,et al.  LITTLE WHITE LIES AND SOCIAL SCIENCE MODELS CORRELATED RESPONSE ERRORS IN A PANEL STUDY OF VOTING , 1992 .

[49]  A. Greenwald,et al.  Attempts to improve the accuracy of self-reports of voting. , 1992 .

[50]  Duane F. Alwin,et al.  INFORMATION TRANSMISSION IN THE SURVEY INTERVIEW: NUMBER OF RESPONSE CATEGORIES AND THE RELIABILITY OF ATTITUDE MEASUREMENT , 1992 .

[51]  Leslie F. Clark,et al.  RATING SCALES NUMERIC VALUES MAY CHANGE THE MEANING OF SCALE LABELS , 1991 .

[52]  Nora Cate Schaeffer,et al.  HARDLY EVER OR CONSTANTLY? GROUP COMPARISONS USING VAGUE QUANTIFIERS , 1991 .

[53]  R. Fay,et al.  An application of Rasch analysis to questionnaire design : using vignettes to study the meaning of work in the Current Population Survey , 1991 .

[54]  Jon A. Krosnick,et al.  The Reliability of Survey Attitude Measurement , 1991 .

[55]  Barbara Means,et al.  When personal history repeats itself: Decomposing memories for recurring events , 1991 .

[56]  Roger Tourangeau,et al.  MEASURING HAPPINESS IN SURVEYS: A TEST OF THE SUBTRACTION HYPOTHESIS , 1991 .

[57]  N. C. Schaeffer,et al.  Two Experiments in Simplifying Response Categories: Intensity Ratings and Behavioral Frequencies , 1991 .

[58]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[59]  G. Willis,et al.  The use of verbal report methods in the development and testing of survey questionnaires , 1991 .

[60]  Scot Burton,et al.  TASK CONDITIONS, RESPONSE FORMULATION PROCESSES, AND RESPONSE ACCURACY FOR BEHAVIORAL FREQUENCY QUESTIONS IN SURVEYS , 1991 .

[61]  Norbert Schwarz,et al.  Assimilation and Contrast Effects in Part-Whole Question Sequences: A Conversational Logic Analysis , 1991 .

[62]  Michaela Wänke,et al.  Semantic and Pragmatic Aspects of Context Effects in Social and Psychological Research , 1991 .

[63]  S. Presser.  CAN CHANGES IN CONTEXT REDUCE VOTE OVERPORTING IN SURVEYS , 1990 .

[64]  Stephen A. Ayidiya,et al.  RESPONSE EFFECTS IN MAIL SURVEYS , 1990 .

[65]  L. Suchman,et al.  Interactional Troubles in Face-to-Face Survey Interviews , 1990 .

[66]  L. Hedges,et al.  Reports of elapsed time: bounding and rounding processes in estimation. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[67]  Barbara Means,et al.  Autobiographical memory for health-related events , 1989 .

[68]  N. C. Schaeffer,et al.  Respondent Behavior in Magnitude Estimation , 1989 .

[69]  M. Yamagishi,et al.  The Influence of Labels and Positions in Rating Scales , 1988 .

[70]  R. Tourangeau,et al.  Cognitive Processes Underlying Context Effects in Attitude Measurement , 1988 .

[71]  N. Mathiowetz,et al.  Out of Work, Out of Mind: Response Errors in Retrospective Reports of Unemployment , 1988 .

[72]  D. Alwin,et al.  The life history calendar: a technique for collecting retrospective data. , 1988, Sociological methodology.

[73]  Norbert Schwarz,et al.  The range of response alternatives may determine the meaning of the question: Further evidence on informative functions of response alternatives. , 1988 .

[74]  E. Blair,et al.  Cognitive Processes Used by Survey Respondents to Answer Behavioral Frequency Questions , 1987 .

[75]  H. Schuman,et al.  Problems in the Use of Survey Questions to Measure Public Opinion , 1987, Science.

[76]  L. Rips,et al.  Answering autobiographical questions: the impact of memory and inference on surveys. , 1987, Science.

[77]  F. Strack,et al.  A comparison of response effects in self-administered and telephone surveys , 1987 .

[78]  S. Presser,et al.  Survey Questions: Handcrafting the Standardized Questionnaire. , 1986 .

[79]  M. Conway,et al.  Remembering one''s own past: The construction of personal histories , 1986 .

[80]  Norbert Schwarz,et al.  Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments , 1985 .

[81]  Lance J Rips,et al.  The subjective dates of natural events in very-long-term memory , 1985, Cognitive Psychology.

[82]  Manfred Kuechler,et al.  Surveying Subjective Phenomena. , 1985 .

[83]  S. Presser,et al.  Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context , 1996 .

[84]  Seymour Sudman,et al.  The Use of Bounded Recall Procedures in Single Interviews , 1984 .

[85]  H. Schuman,et al.  THE NORM OF EVEN-HANDEDNESS IN SURVEYS AS IN LIFE* , 1983 .

[86]  George W. Bohrnstedt,et al.  Chapter 3 – Measurement1 , 1983 .

[87]  W. Belson.  The design and understanding of survey questions , 1982 .

[88]  Norval F. Pohl.  Scale Considerations in Using Vague Quantifiers. , 1981 .

[89]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[90]  Eli P. Cox,et al.  The optimal number of response alternatives for a scale: A review. , 1980 .

[91]  Robert W. Oldendick,et al.  Pseudo-Opinions on Public Affairs , 1980 .

[92]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[93]  F. J. Klopfer,et al.  The Middlemost Choice on Attitude Items , 1980 .

[94]  Robert M. Groves,et al.  Surveys by Telephone: A National Comparison With Personal Interviews. , 1980 .

[95]  Keith S. Dobson,et al.  Equidistant Categorical Labels for Construction of Likert-Type Scales , 1979 .

[96]  N. Bradburn,et al.  How to Ask Questions about Drinking and Sex: Response Effects in Measuring Consumer Behavior , 1976 .

[97]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[98]  M. Jackman.  Education and prejudice or education and response-set? , 1973, American sociological review.

[99]  H. Schuman,et al.  The random probe: a technique for evaluating the validity of closed questions. , 1966, American sociological review.

[100]  J. Neter,et al.  A Study of Response Errors in Expenditures Data from Household Interviews , 1964 .

[101]  N. Cliff,et al.  Adverbs as multipliers. , 1959, Psychological review.

[102]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[103]  Stanley L. Payne,et al.  The Art of Asking Questions , 1951 .