Asking Questions About Behavior: Cognition, Communication, and Questionnaire Construction

Evaluation researchers frequently obtain self-reports of behaviors, asking program participants to report on process and outcome-relevant behaviors. Unfortunately, reporting on one’s behavior poses a difficult cognitive task, and participants’ reports can be profoundly influenced by question wording, format, and context. We review the steps involved in answering a question about one’s behavior and highlight the underlying cognitive and communicative processes. We alert researchers to what can go wrong and provide theoretically grounded recommendations for pilot testing and questionnaire construction.

[1]  S. Withey Reliability of Recall of Income , 1954 .

[2]  A. Barton,et al.  Asking the Embarrassing Question , 1958 .

[3]  A. Parducci Category judgment: a range-frequency model. , 1965, Psychological review.

[4]  W. R. Simmons,et al.  The Unrelated Question Randomized Response Model: Theoretical Framework , 1969 .

[5]  E. E. Jones,et al.  The actor and the observer: Divergent perceptions of the causes of behavior. , 1972 .

[6]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[7]  N. Hatton ASKING QUESTIONS , 1979, The Medical journal of Australia.

[8]  W. Whitten,et al.  Directed search through autobiographical memory , 1981, Memory & cognition.

[9]  W. Belson,et al.  The design and understanding of survey questions , 1982 .

[10]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[11]  John P. Robinson,et al.  Questions and answers in attitude surveys , 1982 .

[12]  D. Watson,et al.  The actor and the observer: How are their perceptions of causality divergent?. , 1982 .

[13]  E F Loftus,et al.  Since the eruption of Mt. St. Helens, has anyone beaten you up? Improving the accuracy of retrospective reports with landmarkevents , 1983, Memory & cognition.

[14]  J. Graham,et al.  Agreement Between Retrospective Accounts of Substance Use and Earlier Reported Substance Use , 1985 .

[15]  E. Loftus,et al.  Retrieving Multiple Autobiographical Memories , 1985 .

[16]  John B. Black,et al.  Knowledge structures in the organization and retrieval of autobiographical memories , 1985, Cognitive Psychology.

[17]  W. Wagenaar My memory: A study of autobiographical memory over six years , 1986, Cognitive Psychology.

[18]  David C. Rubin,et al.  Autobiographical memory: Introduction , 1986 .

[19]  M. Conway,et al.  Remembering one''s own past: The construction of personal histories , 1986 .

[20]  Ulric Neisser,et al.  Nested structure in autobiographical memory. , 1986 .

[21]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[22]  R. Abelson,et al.  Social Information Processing and Survey Methodology. , 1987 .

[23]  Gerhard Strube,et al.  Answering Survey Questions: The Role of Memory , 1987 .

[24]  Roger Tourangeau,et al.  Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. , 1987 .

[25]  E. Blair,et al.  Cognitive Processes Used by Survey Respondents to Answer Behavioral Frequency Questions , 1987 .

[26]  Norbert Schwarz,et al.  The range of response alternatives may determine the meaning of the question: Further evidence on informative functions of response alternatives. , 1988 .

[27]  L. Rips,et al.  Answering autobiographical questions: the impact of memory and inference on surveys. , 1987, Science.

[28]  D. Alwin,et al.  The life history calendar: a technique for collecting retrospective data. , 1988, Sociological methodology.

[29]  N. Mathiowetz,et al.  Out of Work, Out of Mind: Response Errors in Retrospective Reports of Unemployment , 1988 .

[30]  S. Link,et al.  Bias in Quantifying Judgments , 1989 .

[31]  Seymour Sudman,et al.  Contributions of cognitive psychology to advertising research. , 1989 .

[32]  M. Ross Relation of Implicit Theories to the Construction of Personal Histories , 1989 .

[33]  L. Hedges,et al.  Reports of elapsed time: bounding and rounding processes in estimation. , 1990, Journal of experimental psychology. Learning, memory, and cognition.

[34]  A. Baddeley Human Memory: Theory and Practice, Revised Edition , 1990 .

[35]  G. Willis,et al.  The use of verbal report methods in the development and testing of survey questionnaires , 1991 .

[36]  Hans-Jürgen Hippler,et al.  CONFIDENTIALITY ASSURANCES IN SURVEYS: REASSURANCE OR THREAT? , 1992 .

[37]  A. Sanford,et al.  Context Effects and the Communicative Functions of Quantifiers: Implications for Their Use in Attitude Research , 1992 .

[38]  H. H. Clark,et al.  Asking questions and influencing answers. , 1992 .

[39]  N. Schwarz,et al.  Context Effects in Social and Psychological Research , 1992 .

[40]  G. Menon,et al.  The Effects of Accessibility of Information in Memory on Judgments of Behavioral Frequencies , 1993 .

[41]  Seymour Sudman,et al.  Autobiographical memory and the validity of retrospective reports , 1994 .

[42]  Norbert Schwarz,et al.  Judgment in a Social Context: Biases, Shortcomings, and the Logic of Conversation , 1994 .

[43]  Constantine G. Lyketsos,et al.  The Life Chart Interview: A standardized method to describe the course of psychopathology. , 1994 .

[44]  Larry V. Hedges,et al.  Telescoping and Temporal Memory , 1994 .

[45]  Self and Proxy Reports of Everyday Events , 1994 .

[46]  Paul Beatty,et al.  RESPONSE RATES AND RESPONSE CONTENT IN MAIL VERSUS FACE-TO-FACE SURVEYS , 1994 .

[47]  Geeta Menon Judgments of Behavioral Frequencies: Memory Search and Retrieval Strategies , 1994 .

[48]  K. Fiedler,et al.  Two halfs may be more than one whole : category-split effects on frequency illusions , 1994 .

[49]  Norman M. Bradburn,et al.  DO RESPONDENTS REALLY “MARK ALL THAT APPLY” ON SELF-ADMINISTERED QUESTIONS? , 1994 .

[50]  Daniel B. Wright,et al.  SURVEY QUESTIONS ABOUT THE FREQUENCY OF VAGUELY DEFINED EVENTS: THE EFFECTS OF RESPONSE ALTERNATIVES , 1994 .

[51]  The Impact of Differing Memory Domains on Event-Dating Processes in Self and Proxy Reports , 1994 .

[52]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[53]  N. Schwarz,et al.  BEHAVIORAL FREQUENCY JUDGMENTS - AN ACCESSIBILITY DIAGNOSTICITY FRAMEWORK , 1995 .

[54]  Judith T. Lessler,et al.  A coding system for appraising questionnaires. , 1996 .

[55]  Theresa J. DeMaio,et al.  Cognitive interviewing techniques: In the lab and in the field. , 1996 .

[56]  Avshalom Caspi,et al.  THE LIFE HISTORY CALENDAR: A RESEARCH AND CLINICAL ASSESSMENT METHOD FOR COLLECTING RETROSPECTIVE EVENT‐HISTORY DATA , 1996 .

[57]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[58]  M. Schober,et al.  Does conversational interviewing reduce survey measurement error?. Public opinion quarterly, . , 1997 .

[59]  Frederick G. Conrad,et al.  Does Conversational Interviewing Reduce Survey Measurement Error , 1997 .

[60]  Don A. Dillman,et al.  Towards a Theory of Self‐Administered Questionnaire Design , 1997 .

[61]  R. Belli,et al.  The structure of autobiographical memory and the event history calendar: potential improvements in the quality of retrospective reports in surveys. , 1998, Memory.

[62]  Julie. Barnett,et al.  Asking sensitive questions. , 1998 .

[63]  N. Schwarz,et al.  Looking back at anger: reference periods change the interpretation of emotion frequency questions. , 1998, Journal of personality and social psychology.

[64]  N. Schwarz,et al.  Cognitive Dynamics of Proxy Responding: The Diverging Perspec , 1998 .

[65]  Bärbel Knäuper,et al.  The Impact of Age and Education on Response Order Effects in Attitude Measurement , 1999 .

[66]  William G. Axinn,et al.  Innovations in life history calendar applications , 1999 .

[67]  Norbert Schwarz,et al.  Telling what they want to know: participants tailor causal attributions to researchers' interests , 1999 .

[68]  木村 和夫 Pragmatics , 1997, Language Teaching.

[69]  N. Schwarz Self-reports: How the questions shape the answers. , 1999 .

[70]  R. Nisbett,et al.  Culture, Autobiographical Memory, and Behavioral Frequency Reports: Measurement Issues in Cross-Cultural Studies , 2000 .

[71]  Roger Tourangeau,et al.  Cognition and survey research , 2000 .

[72]  Alexander J. Rothman,et al.  "How Many Partners Is Too Many?" Shaping Perceptions of Personal Vulnerability' , 2001 .

[73]  N. Brown Encoding, representing, and estimating event frequencies: a multiple strategy perspective , 2002 .

[74]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[75]  S. Bhatti,et al.  Questions, questions. , 2002, British dental journal.