Mode system effects in an online panel study : Comparing a probability-based online panel with two face-to-face reference surveys

One of the methods for evaluating online panels in terms of data quality is comparing the estimates that the panels provide with benchmark sources. For probability-based online panels, high-quality surveys or government statistics can be used as references. If differences among the benchmark and the online panel estimates are found, these can have several causes. First, the question wordings can differ between the sources, which can lead to differences in measurement. Second, the reference and the online panel may not be comparable in terms of sample composition. Finally, since the reference estimates are usually collected face-to-face or by telephone, mode effects might be expected. In this article, we investigate mode system effects, an alternative to mode effects that does not focus solely on measurement differences between the modes, but also incorporates survey design features into the comparison. The data from a probability-based offline-recruited online panel is compared to the data from two face-to-face surveys with almost identical recruitment protocols. In the analysis, the distinction is made between factual and attitudinal questions. We report both effect sizes of the differences and significances. The results show that the online panel differs from face-to-face surveys in both attitudinal and factual measures. However, the reference surveys only differ in attitudinal measures and show no significant differences for factual questions. We attribute this to the instability of attitudes and thus show the importance of triangulation and using two surveys of the same mode for comparison.

[1]  Michael Blohm,et al.  Konzeption und Durchführung der "Allgemeinen Bevölkerungsumfrage der Sozialwissenschaften" (ALLBUS) 2004 , 1996 .

[2]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[3]  Siegfried Gabler,et al.  Weighting for Unequal Inclusion Probabilities and Nonresponse in Dual Frame Telephone Surveys , 2011 .

[4]  R. Groves,et al.  Survey Errors and Survey Costs. , 1991 .

[5]  D. Leeuw,et al.  Data Quality in Mail, Telephone and Face to Face Surveys. , 1992 .

[6]  Matthias Schonlau,et al.  Noncoverage and nonresponse in an Internet survey , 2007 .

[7]  R. Little,et al.  Models for Contingency Tables with Known Margins when Target and Sampled Populations Differ , 1991 .

[8]  Lars E. Lyberg,et al.  Data Collection Methods and Measurement Error: An Overview , 2011 .

[9]  E. Ziegel,et al.  Nonresponse In Household Interview Surveys , 1998 .

[10]  Roger Tourangeau,et al.  The Science of Web Surveys , 2013 .

[11]  Paul P. Biemer,et al.  Introduction to Survey Quality , 2003 .

[12]  Barry Schouten,et al.  Disentangling mode-specific selection and measurement bias in social surveys. , 2013, Social science research.

[13]  J. Hox,et al.  Tilburg University Measurement equivalence in mixed mode surveys , 2015 .

[14]  Daniel L. Oberski,et al.  Understanding Society Innovation Panel Wave 9 , 2017 .

[15]  M. Couper,et al.  Sample Composition Discrepancies in Different Stages of a Probability-based Online Panel , 2013 .

[16]  Eine allgemeine Formel zur Anpassung an Randtabellen , 1994 .

[17]  Peter Stopher.  Collecting, Managing, and Assessing Data Using Sample Surveys: Measuring data quality , 2012 .

[18]  C. Ferguson.  An effect size primer: A guide for clinicians and researchers. , 2009 .

[19]  E. Leeuw,et al.  To mix or not to mix data collection modes in surveys. , 2005 .

[20]  Coverage- und Nonresponse-Effekte bei Online-Bevölkerungsumfragen , 2009 .

[21]  Mick P. Couper,et al.  The Future of Modes of Data Collection , 2011 .

[22]  W. Deming,et al.  On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known , 1940 .

[23]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[24]  Barry Schouten,et al.  Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions , 2013 .

[25]  Sunghee Lee.  An Evaluation of Nonresponse and Coverage Errors in a Prerecruited Probability Web Panel Survey , 2006 .

[26]  Mario Callegaro,et al.  A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples1 , 2014 .