Using paradata to explore item level response times in surveys

Summary.  We analyse item level keystroke data from cycle 6 of the National Survey of Family Growth, which is a survey on fertility and related topics that is conducted in the USA. The National Survey of Family Growth is conducted among both males and females by using computer‐assisted personal visit interviews and an audio computer‐assisted self‐interviewing component for the most sensitive topics. Our analyses focus on the time taken to answer a question as a function of item level characteristics, respondent characteristics and interviewer characteristics. Using multilevel models, we explore how these factors influence response times. Our exploratory study suggests that factors at all three levels (item, respondent and interviewer) influence response times. These results demonstrate that question features that explain variation in response times can be automatically derived from standard computer‐assisted personal interviewing paradata. The effects of respondent characteristics that we observe are in line with prior findings from more controlled studies conducted in supervised telephone facilities. Some demographic characteristics of interviewers contributed to the variation in response times, though they failed to explain large portions of the between‐interviewer variance.

[1]  P. Deb.  Finite Mixture Models , 2008 .

[2]  Ayanna K. Thomas,et al.  Theoretical Perspectives on Cognitive Aging , 2019, Handbook of Medical Neuropsychology.

[3]  R. Rindfuss,et al.  Do low survey response rates bias results? Evidence from Japan , 2015 .

[4]  R. Winkelmann,et al.  Reported Happiness, Fast and Slow , 2012 .

[5]  To Agree or Not to Agree ? Impact of interviewer speech on survey participation decisions , 2010 .

[6]  Frauke Kreuter,et al.  The use of paradata to monitor and manage survey data collection , 2010 .

[7]  Roger Tourangeau,et al.  Taking the Audio Out of Audio-CASI , 2009 .

[8]  Frauke Kreuter,et al.  Analyzing contact sequences in call record data. Potential and limitations of sequence indicators for nonresponse adjustments in the European Social Survey , 2009 .

[9]  M. Deutschmann.  Rezension: Jochen Mayerl und Dieter Urban, 2008: Antwortreaktionszeiten in Survey-Analysen , 2009 .

[10]  Michael J. Stern,et al.  The Use of Client-side Paradata in Analyzing the Effects of Visual Layout on Changing Responses in Web Surveys , 2008 .

[11]  R. Tourangeau,et al.  Fast times and easy questions: the effects of age, experience and question complexity on web survey response times , 2008 .

[12]  François Laflamme.  Data Collection Research using Paradata at Statistics Canada , 2008 .

[13]  Andy Peytchev,et al.  Effect of Interviewer Experience on Interview Pace and Interviewer Attitudes , 2007 .

[14]  Brady T. West,et al.  Linear Mixed Models: A Practical Guide Using Statistical Software , 2006 .

[15]  Karen E Davis,et al.  National Survey of Family Growth, Cycle 6: sample design, weighting, imputation, and variance estimation. , 2006, Vital and health statistics. Series 2, Data evaluation and methods research.

[16]  Robert M. Groves,et al.  Responsive design for household surveys: tools for actively controlling survey errors and costs , 2006 .

[17]  F. Conrad,et al.  Interactive Feedback Can Improve the Quality of Responses in Web Surveys , 2005 .

[18]  Piet Sellke,et al.  Analyzing cognitive processes in CATI-surveys with response latencies : an empirical evaluation of the consequences of using different baseline speed measures , 2005 .

[19]  W. Dijkstra,et al.  Response latencies and (para)linguistic expressions as indicators of response error , 2004 .

[20]  Elizabeth Martin,et al.  METHODS FOR TESTING AND EVALUATING SURVEY QUESTIONS , 2004 .

[21]  Dirk Heerwegh,et al.  Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey , 2003 .

[22]  J. N. Bassili,et al.  The minority slowness effect: subtle inhibitions in the expression of views not shared by others. , 2003, Journal of personality and social psychology.

[23]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[24]  Rachel A. Caspar.  USING KEYSTROKE FILES TO ASSESS RESPONDENT DIFFICULTIES WITH AN AUDIO-CASI INSTRUMENT , 2002 .

[25]  Mick P. Couper,et al.  USING TRACE FILES TO EVALUATE THE NATIONAL HEALTH INTERVIEW SURVEY CAPI INSTRUMENT , 2002 .

[26]  Mick P. Couper,et al.  MEASURING SURVEY QUALITY IN A CASIC ENVIRONMENT , 2002 .

[27]  M. Wagner-Menghin.  Towards the Identification of Non-Scalable Personality Questionnaire Respondents: Taking Response Time into Account , 2002 .

[28]  Geoffrey J. McLachlan,et al.  Finite Mixture Models , 2019, Annual Review of Statistics and Its Application.

[29]  Jon A. Krosnick,et al.  Do Strength‐Related Attitude Properties Determine Susceptibility to Response Effects? New Evidence From Response Latency, Attitude Extremity, and Aggregate Indices , 2000 .

[30]  D. Balota,et al.  Individual differences in information-processing rate and amount: implications for group differences in response latency. , 1999, Psychological bulletin.

[31]  E. S. Knowles,et al.  Why people say "yes": A dual-process theory of acquiescence. , 1999 .

[32]  Seymour Sudman,et al.  Cognition, aging, and self-reports , 1999 .

[33]  Jeffrey Levine,et al.  Election Campaigns, Social Communication, and the Accessibility of Perceived Discussant Preference , 1998 .

[34]  Sue Ellen Hansen,et al.  Evaluating Interviewer Use of CAPI Technology , 1997 .

[35]  D. Fisher,et al.  Molar and latent models of cognitive slowing: Implications for aging, dementia, depression, development, and intelligence , 1996, Psychonomic bulletin & review.

[36]  Eliot R. Smith,et al.  Accessible attitudes influence categorization of multiply categorizable objects. , 1996, Journal of personality and social psychology.

[37]  J. N. Bassili,et al.  RESPONSE LATENCY AS A SIGNAL TO QUESTION PROBLEMS IN SURVEY RESEARCH , 1996 .

[38]  I. Ajzen,et al.  Identifying Salient Beliefs About Leisure Activities: Frequency of Elicitation Versus Response Latency1 , 1995 .

[39]  R. Tourangeau.  Context Effects on Responses to Attitude Questions: Attitudes as Memory Structures , 1992 .

[40]  Russell H. Fazio,et al.  A practical guide to the use of response latency in social psychological research. , 1990 .

[41]  R. Fazio,et al.  Attitude accessibility as a moderator of the attitude-perception and attitude-behavior relations: an investigation of the 1984 presidential election. , 1986, Journal of personality and social psychology.

[42]  Eliot R. Smith,et al.  Development of automatism of social judgments. , 1986 .

[43]  R. Fazio.  How do attitudes guide behavior , 1986 .

[44]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[45]  Shirley A. Star,et al.  AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH , 1980 .

[46]  E. Smith,et al.  Choice reaction time: an analysis of the major theoretical positions. , 1968, Psychological bulletin.