Disentangling mode-specific selection and measurement bias in social surveys.

A large-scale mixed-mode experiment linked to the Dutch Crime Victimization Survey was conducted in 2011. The experiment consisted of two waves; one wave with random assignment to one of the modes web, paper, telephone and face-to-face, and one follow-up wave to the full sample with interviewer modes only. The objective of the experiment is to estimate total mode effects and more specifically the corresponding mode effect components arising from undercoverage, nonresponse and measurement. In this paper, mode-specific selection and measurement bias are defined, and estimators for the bias terms based on the experimental design are introduced and discussed. The proposed estimators are applied to a number of key survey variables from the Labour Force Survey and the Crime Victimization Survey.

[1]  Melanie C. Green,et al.  Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias , 2003 .

[2]  F. Conrad,et al.  Spacing, Position, and Order Interpretive Heuristics for Visual Features of Survey Questions , 2004 .

[3]  Robert D. Tortora,et al.  Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet , 2009 .

[4]  R. Little,et al.  Inference for the Population Total from Probability-Proportional-to-Size Samples Based on Predictions from a Penalized Spline Nonparametric Model , 2003 .

[5]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[6]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[7]  Don A. Dillman,et al.  Survey Mode as a Source of Instability in Responses across Surveys , 2005 .

[8]  Margo J. Anderson,et al.  Challenges to the confidentiality of U.S. Federal statistics, 1910-1965 , 2007 .

[9]  E. Leeuw,et al.  To mix or not to mix data collection modes in surveys. , 2005 .

[10]  Paul Beatty,et al.  RESPONSE RATES AND RESPONSE CONTENT IN MAIL VERSUS FACE-TO-FACE SURVEYS , 1994 .

[11]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[12]  Paul Biemer,et al.  Nonresponse Bias and Measurement Bias in a Comparison of Face to Face and Telephone Interviewing , 2001 .

[13]  E. Roberts Caroline,et al.  Mixing modes of data collection in surveys: A methodological review , 2007 .

[14]  B. Schouten.  A selection strategy for weighting variables under a not-missing-at-random assumption , 2007 .

[15]  Geert Molenberghs,et al.  A Method for Evaluating Mode Effects in Mixed-mode Surveys , 2010 .

[16]  Shirley Dex,et al.  On the experience and evidence about mixing modes of data collection in large-scale surveys where the web is used as one of the modes in data collection , 2011 .

[17]  Jolene D. Smyth,et al.  The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys , 2007 .

[18]  Carl-Erik Särndal,et al.  Model Assisted Survey Sampling , 1997 .

[19]  G. Molenberghs,et al.  A Method to Evaluate Mode Effects on the Mean and Variance of a Continuous Variable in Mixed‐Mode Surveys , 2012 .

[20]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[21]  Geert Loosveldt,et al.  Evaluating Relative Mode Effects in Mixed-Mode Surveys: , 2013 .

[22]  F. Conrad,et al.  Color, Labels, and Interpretive Heuristics for Response Scales , 2007 .

[23]  Peter Lynn,et al.  Assessing the Effect of Data Collection Mode on Measurement , 2010 .