Risk of Nonresponse Bias and the Length of the Field Period in a Mixed-Mode General Population Panel

Survey researchers are often confronted with the question of how long to set the length of the field period. Longer fielding time might lead to greater participation yet requires survey managers to devote more of their time to data collection efforts. With the aim of facilitating the decision about the length of the field period, we investigated whether a longer fielding time reduces the risk of nonresponse bias to judge whether field periods can be ended earlier without endangering the performance of the survey. By using data from six waves of a probability-based mixed-mode (online and mail) panel of the German population, we analyzed whether the risk of nonresponse bias decreases over the field period by investigating how day-by-day coefficients of variation develop during the field period. We then determined the optimal cut-off points for each mode after which data collection can be terminated without increasing the risk of nonresponse bias and found that the optimal cut-off points differ by mode. Our study complements prior research by shifting the perspective in the investigation of the risk of nonresponse bias to panel data as well as to mixed-mode surveys, in particular. Our proposed method of using coefficients of variation to assess whether the risk of nonresponse bias decreases significantly with each additional day of fieldwork can aid survey practitioners in finding the optimal field period for their mixed-mode surveys.

[1]  M. N. Donald,et al.  IMPLICATIONS OF NONRESPONSE FOR THE INTERPRETATION OF MAIL QUESTIONNAIRE DATA , 1960 .

[2]  Taylor H Lewis.  Multivariate Tests for Phase Capacity , 2019 .

[3]  Marcel Das,et al.  Innovation in Online Data Collection for Scientific Research: The Dutch MESS Project , 2012 .

[4]  Juan Sánchez-Fernández,et al.  Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings , 2010 .

[5]  J. Whitehead,et al.  Sample Non-response Bias and Aggregate Benefits in Contingent Valuation: an Examination of Early, Late and Non-respondents , 1993 .

[6]  Patrick Sturgis,et al.  Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis , 2017 .

[7]  Jelke Bethlehem,et al.  Indicators for the representativeness of survey response , 2009 .

[8]  Robert M. Groves,et al.  Responsive design for household surveys: tools for actively controlling survey errors and costs , 2006 .

[9]  P. Lugtig.  Panel Attrition , 2014 .

[10]  Modeling the Weekly Data Collection Efficiency of Face-to-Face Surveys: Six Rounds of the European Social Survey , 2017 .

[11]  Frauke Kreuter,et al.  Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data , 2010 .

[12]  T. Koepsell,et al.  Characteristics of telephone survey respondents according to willingness to participate. , 2003, American journal of epidemiology.

[13]  James Wagner,et al.  Does more balanced survey response imply less non‐response bias? , 2016 .

[14]  Bella Struminskaya,et al.  Establishing an Open Probability-Based Mixed-Mode Panel of the General Population in Germany , 2018 .

[15]  Jennie Connor,et al.  Non-response bias in a web-based health behaviour survey of New Zealand tertiary students. , 2011, Preventive medicine.

[16]  Olena Kaminska,et al.  Satisficing Among Reluctant Respondents in a Cross-National Context , 2010 .

[17]  Frauke Kreuter,et al.  A Note on Mechanisms Leading to Lower Data Quality of Late or Reluctant Respondents , 2014 .

[18]  Robert M. Groves,et al.  The Impact of Nonresponse Rates on Nonresponse Bias A Meta-Analysis , 2008 .

[19]  Paul P. Biemer,et al.  Introduction to Survey Quality , 2003 .

[20]  Martin Wetzels,et al.  Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study , 2004 .

[21]  Laith Alattar,et al.  An Introduction to the Understanding America Study Internet Panel , 2018 .

[22]  Lars Kaczmirek,et al.  Mode system effects in an online panel study : Comparing a probability-based online panel with two face-to-face reference surveys , 2015 .

[23]  Kypros Kypri,et al.  Assessment of nonresponse bias in an internet survey of alcohol use. , 2004, Alcoholism, clinical and experimental research.

[24]  Eager, Reluctant, and Nonresponders to a Mailed Longitudinal Survey: Attitudinal and Substance Use Characteristics Differentiate Respondents1 , 1998 .

[25]  Peter J. Smith,et al.  Data set representativeness during data collection in three UK social surveys: generalizability and the effects of auxiliary covariate choice , 2018 .

[26]  Ricky Telg,et al.  Choosing to Use the Web: Comparing Early and Late Respondents to an Online Web- based Survey Designed to Assess IT Computer Skills Perceptions of County Extension Agents , 2004 .

[27]  Vidal Díaz de Rada,et al.  The Effect of Follow-up Mailings on The Response Rate and Response Quality in Mail Surveys , 2005 .

[28]  Natalie Shlomo,et al.  Estimation of an indicator of the representativeness of survey response , 2012 .

[29]  Randall J. Olsen,et al.  The problem of respondent attrition: survey methodology is key , 2005 .

[30]  James Wagner,et al.  A Comparison of Alternative Indicators for the Risk of Nonresponse Bias. , 2012, Public opinion quarterly.

[31]  R. Little,et al.  Does Weighting for Nonresponse Increase the Variance of Survey Means? (Conference Paper) , 2004 .

[32]  Should the Third Reminder be Sent? The Role of Survey Response Timing on Web Survey Results , 2013 .

[33]  Nicole Watson,et al.  Identifying Factors Affecting Longitudinal Survey Response , 2009 .

[34]  Bella Struminskaya,et al.  Early and Late Participation during the Field Period: Response Timing in a Mixed-Mode Probability-Based Panel Survey , 2020, Sociological Methods & Research.

[35]  G. Buhyoff,et al.  Mailed questionnaire surveys and the reluctant respondent: an empirical examination of differences between early and late respondents. , 1980 .

[36]  Joseph W. Sakshaug,et al.  Does the Recruitment of Offline Households Increase the Sample Representativeness of Probability-Based Online Panels? Evidence From the German Internet Panel , 2017 .

[37]  Taylor H. Lewis,et al.  Univariate Tests for Phase Capacity: Tools for Identifying When to Modify a Survey’s Data Collection Protocol , 2017 .

[38]  E. Singer,et al.  The effects of response rate changes on the index of consumer sentiment. , 2000, Public opinion quarterly.

[39]  G. Loosveldt,et al.  Fieldwork Monitoring for the European Social Survey: An illustration with Belgium and the Czech Republic in Round 7 , 2017 .

[40]  Robert M. Groves,et al.  Nonresponse in Household Interview Surveys: Groves/Nonresponse , 1998 .

[41]  T. Heberlein,et al.  Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. , 1978 .

[42]  Richard S. Sigman,et al.  Does the Length of Fielding Period Matter? Examining Response Scores of Early Versus Late Responders , 2014 .

[43]  A. Peytchev,et al.  Not All Survey Effort is Equal: Reduction of Nonresponse Bias and Nonresponse Error , 2009 .