Response Error in a Web Survey and a Mailed Questionnaire: The Role of Cognitive Functioning

Web-based interviewing is gradually replacing traditional modes of data collection, in particular telephone and mailed surveys. This global trend takes place despite the fact that established knowledge of its consequences on response error is incomplete. This paper studies differences between a web (CAWI) and a mailed version (MAIL) of a questionnaire in various forms of response error, namely item nonresponse, satisficing, person-reliability, and social desirable responding. We posit 1) that response error depends on respondents cognitive functioning, namely in the domains of global reading abilities, fluid intelligence, as well as working and episodic memory; and 2) that these effects differ across modes of data collection with generally higher prevalence in the CAWI mode since this mode is more demanding. The analysis builds on a randomized mode experiment implemented in the context of the Berlin Aging Study II (BASE-II), a survey that primarily focuses on multidimensional processes of physical and mental aging (see Bertram et al. 2014). The analysis reveals a high impact of cognitive functioning at the various stages of the survey response process. While we do found moderate mode-differences in response error, such as higher item nonresponse rates in the CAWI mode, we did not find cognitive functioning to be a better predictor of response error in web-based interviewing.

[1]  R B Wallace,et al.  Data quality and age: health and psychobehavioral correlates of item nonresponse and inconsistent responses. , 1989, Journal of gerontology.

[2]  E. Roberts Caroline,et al.  Mixing modes of data collection in surveys: A methodological review , 2007 .

[3]  Gert G. Wagner,et al.  The Socio-Economic Module of the Berlin Aging Study II (SOEP-BASE): Description, Structure, and Questionnaire , 2013 .

[4]  D. Leeuw,et al.  Data Quality in Mail, Telephone and Face to Face Surveys. , 1992 .

[5]  T. Salthouse.  Anticipatory processing in transcription typing. , 1985, The Journal of applied psychology.

[6]  D. Paulhus.  Two-component models of socially desirable responding. , 1984 .

[7]  M. Mather.  The Affective Neuroscience of Aging. , 2016, Annual review of psychology.

[8]  H. Wimmer,et al.  A dual-route perspective on poor reading in a regular orthography: Evidence from phonological and orthographic lexical decisions , 2008, Cognitive neuropsychology.

[9]  Anne Morris,et al.  E-literacy and the grey digital divide: a review with recommendations , 2007 .

[10]  D. Wechsler.  Manual for the Wechsler Adult Intelligence Scale. , 1955 .

[11]  Don A. Dillman,et al.  Development of a Standard E-Mail Methodology: Results of an Experiment , 1998 .

[12]  Klaas Sijtsma,et al.  Methodology Review: Evaluating Person Fit , 2001 .

[13]  Zheng Yan,et al.  Factors affecting response rates of the web survey: A systematic review , 2010, Comput. Hum. Behav..

[14]  P. Carpenter,et al.  Individual differences in working memory and reading , 1980 .

[15]  Seymour Sudman,et al.  Response Effects in Surveys: A Review and Synthesis , 1974 .

[16]  Jon A. Krosnick,et al.  Satisficing in surveys: Initial evidence , 1996 .

[17]  Troy Devon Thomas,et al.  Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies , 2013 .

[18]  Shu-Chen Li,et al.  Cohort profile: The Berlin Aging Study II (BASE-II). , 2014, International journal of epidemiology.

[19]  R. Schulz,et al.  A life span model of successful aging. , 1996, The American psychologist.

[20]  Matthias Schonlau,et al.  Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature , 2002 .

[21]  Gert G. Wagner,et al.  Selectivity Processes in and Weights for the Berlin Aging Study II (Base-II) , 2013 .

[22]  R. Cattell.  ESTIMATING MODULATOR INDICES AND STATE LIABILITIES. , 1971, Multivariate behavioral research.

[23]  Tse-Hua Shih,et al.  Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis , 2008 .

[24]  P. Chisnall.  Mail and Internet Surveys: The Tailored Design Method , 2007, Journal of Advertising Research.

[25]  U. Lindenberger,et al.  Frontiers in Aging Neuroscience Aging Neuroscience Hundred Days of Cognitive Training Enhance Broad Cognitive Abilities in Adulthood: Findings from the Cogito Study Materials and Methods Participants and Procedure , 2022 .

[26]  George Karabatsos,et al.  Comparing the Aberrant Response Detection Performance of Thirty-Six Person-Fit Statistics , 2003 .

[27]  Kenneth K. Boyer,et al.  Print versus electronic surveys: A comparison of two data collection methodologies , 2002 .

[28]  Roger Tourangeau,et al.  The Science of Web Surveys , 2013 .

[29]  Weiwei Lin,et al.  Web and Mail Surveys , 2012 .

[30]  Matthias Schonlau,et al.  Noncoverage and nonresponse in an Internet survey , 2007 .

[31]  Judith M. Conijn.  Detecting and explaining person misfit in non-cognitive measurement , 2013 .

[32]  Beatrice Rammstedt,et al.  Cognitive ability, acquiescence, and the structure of personality in a sample of older adults. , 2015, Psychological assessment.

[33]  S. Presser,et al.  Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context , 1996 .

[34]  A. Baddeley.  Working memory: theories, models, and controversies. , 2012, Annual review of psychology.

[35]  Mick P. Couper,et al.  The Future of Modes of Data Collection , 2011 .

[36]  S. D. Lima,et al.  General slowing in semantic priming and word recognition. , 1992, Psychology and aging.

[37]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[38]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[39]  R Kliegl,et al.  Speed and intelligence in old age. , 1993, Psychology and aging.

[40]  E. Tulving,et al.  Episodic and semantic memory , 1972 .

[41]  S. Kiesler,et al.  Response Effects in the Electronic Survey , 1986 .

[42]  J. Schupp,et al.  Using Person-Fit Measures to Assess the Impact of Panel Conditioning on Reliability , 2016 .

[43]  Roger Tourangeau,et al.  Survey research and societal change. , 2004, Annual review of psychology.

[44]  E. Tulving.  [Episodic memory: from mind to brain]. , 2004, Revue neurologique.

[45]  M. Scheier,et al.  Personality and quality of life: The importance of optimism and goal adjustment , 2004, Quality of Life Research.

[46]  C. Grady.  Cognitive Neuroscience of Aging , 2008, Annals of the New York Academy of Sciences.

[47]  K. Stanovich.  Matthew Effects in Reading: Some Consequences of Individual Differences in the Acquisition of Literacy , 2009 .

[48]  Gert G. Wagner,et al.  The German Socio-Economic Panel Study (SOEP) - Scope, Evolution and Enhancements , 2007, Journal of Contextual Economics – Schmollers Jahrbuch.

[49]  Paul P. Biemer,et al.  Introduction to Survey Quality , 2003 .

[50]  Kevin B. Wright,et al.  Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire Authoring Software Packages, and Web Survey Services , 2006, J. Comput. Mediat. Commun..

[51]  Don A. Dillman,et al.  Do Mail and Internet Surveys Produce Different Item Nonresponse Rates? An Experiment Using Random Mode Assignment , 2012 .

[52]  H. Wimmer,et al.  No deficits at the point of hemispheric indecision , 2002, Neuropsychologia.

[53]  Zvia Breznitz,et al.  Increasing first graders' reading accuracy and comprehension by accelerating their reading rates. , 1987 .

[54]  S. Jay Samuels,et al.  Toward a theory of automatic information processing in reading , 1974 .

[55]  Sara J. Czaja,et al.  Computer Technology and the Older Adult , 1997 .

[56]  J. Connell,et al.  Individual differences and the development of perceived control. , 1998, Monographs of the Society for Research in Child Development.

[57]  Shu Tian Cole.  Comparing Mail and Web-Based Survey Distribution Methods: Results of Surveys to Leisure Travel Retailers , 2005 .

[58]  Timothy V. Rasinski,et al.  Speed Does Matter in Reading. , 2000 .

[59]  Ángel Borrego,et al.  Response Rates and Data Quality in Web and Mail Surveys Administered to PhD Holders , 2011 .

[60]  U. Lindenberger,et al.  The Subjective Health Horizon Questionnaire (SHH-Q): Assessing Future Time Perspectives for Facets of an Active Lifestyle , 2016, Gerontology.

[61]  Person fit for tests with polytomous responses , 2005 .

[62]  Bengü Börkan,et al.  The Mode Effect in Mixed-Mode Surveys , 2010 .

[63]  K. Motamedi.  The quality of American life , 1976 .

[64]  C. Ross,et al.  Socially-desirable response and acquiescence in a cross-cultural survey of mental health. , 1984, Journal of health and social behavior.

[65]  M. Fuchs.  Item-Nonresponse in einer Befragung von Alten und Hochbetagten , 2009 .

[66]  John P. Robinson,et al.  Questions and answers in attitude surveys , 1982 .

[67]  Rob R. Meijer,et al.  Detection of Person Misfit in Computerized Adaptive Tests with Polytomous Items , 2002 .

[68]  R. Klassen,et al.  Experimental comparison of Web, electronic and mail survey technologies in operations management , 2001 .