Fast times and easy questions: the effects of age, experience and question complexity on web survey response times

This paper examines response times (RT) to survey questions. Cognitive psychologists have long relied on response times to study cognitive processes but response time data have only recently received attention from survey researchers. To date, most of the studies on response times in surveys have treated response times either as a predictor or as a proxy measure for some other variable (e.g. attitude accessibility) of greater interest. As a result, response times have not been the main focus of the research. Focusing on the nature and determinants of response times, this paper examines variables that affect how long it takes respondents to answer questions in web surveys. Using the survey response model proposed by Tourangeau, Rips, and Rasinski (2000), we include both item-level characteristics and respondent-level characteristics thought to affect response times in a two-level cross-classified model. Much of the time spent on processing the questions involves reading and interpreting them. The results from the cross-classified models indicate that response times are affected by question characteristics such as the total number of clauses and the number of words per clause that probably reflect reading times. In addition, response times are also affected by the number and type of answer categories, and the location of the question within the questionnaire, as well as respondent characteristics such as age, education and experience with the Internet and with completing web surveys. Aside from their fixed effects on response times, respondent-level characteristics (such as age) are shown to vary randomly over questions and effects of question-level characteristics (such as types of questions and response scales) vary randomly over respondents. Copyright © 2007 John Wiley & Sons, Ltd.

[1]  Scott Fricker,et al.  An Experimental Comparison of Web and Telephone Surveys , 2005 .

[2]  J. Gill.  Hierarchical Linear Models , 2005 .

[3]  F. Conrad,et al.  Spacing, Position, and Order Interpretive Heuristics for Visual Features of Survey Questions , 2004 .

[4]  W. Dijkstra,et al.  Response latencies and (para)linguistic expressions as indicators of response error , 2004 .

[5]  N. Schwarz,et al.  Thinking About Answers: The Application of Cognitive Processes to Survey Methodology , 1995, Quality of Life Research.

[6]  Dirk Heerwegh,et al.  Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey , 2003 .

[7]  Kenneth Mulligan,et al.  Response Latency Methodology for Survey Research: Measurement and Modeling Strategies , 2003, Political Analysis.

[8]  T. Zandt.  Analysis of Response Time Distributions , 2002 .

[9]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[10]  J. N. Bassili,et al.  Editor's Introduction: Reflections on Response Latency Measurement in Telephone Surveys , 2000 .

[11]  M. Couper.  A REVIEW OF ISSUES AND APPROACHES , 2000 .

[12]  Seymour Sudman,et al.  Cognition, aging, and self-reports , 1999 .

[13]  J. N. Bassili,et al.  RESPONSE LATENCY AS A SIGNAL TO QUESTION PROBLEMS IN SURVEY RESEARCH , 1996 .

[14]  Seymour Sudman,et al.  Thinking about Answers: The Application of Cognitive Process to Survey Methodology , 1996 .

[15]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[16]  R. Ratcliff.  Methods for dealing with reaction time outliers. , 1993, Psychological bulletin.

[17]  J. N. Bassili,et al.  RESPONSE LATENCY VERSUS CERTAINTY AS INDEXES OF THE STRENGTH OF VOTING INTENTIONS IN A CATI SURVEY , 1993 .

[18]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[19]  M. Just,et al.  From the SelectedWorks of Marcel Adam Just 1992 A capacity theory of comprehension : Individual differences in working memory , 2017 .

[20]  Russell H. Fazio,et al.  New technologies for the direct and indirect assessment of attitudes. , 1992 .

[21]  J. N. Bassili,et al.  Response-time measurement in survey research: A method for CATI and a new look at nonattitudes. , 1991 .

[22]  Roger Tourangeau,et al.  Attitude structure and belief accessibility , 1991 .

[23]  R. Fazio.  Multiple Processes by which Attitudes Guide Behavior: The Mode Model as an Integrative Framework , 1990 .

[24]  F. Strack,et al.  Thinking, Judging, and Communicating: A Process Account of Context Effects in Attitude Surveys , 1987 .

[25]  R. Fazio,et al.  On the automatic activation of attitudes. , 1986, Journal of personality and social psychology.

[26]  Shirley A. Star,et al.  AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH , 1980 .

[27]  H. H. Clark.  The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. , 1973 .

[28]  P. Converse,et al.  Attitudes and Non-attitudes: Continuation of a Dialogue , 1970 .

[29]  F. Donders,et al.  Over de snelheid van psychische Processen , 1868 .