Measurement Error Calibration in Mixed-mode Sample Surveys

Mixed-mode surveys are known to be susceptible to mode-dependent selection and measurement effects, collectively referred to as mode effects. The use of different data collection modes within the same survey may reduce selectivity of the overall response but is characterized by measurement errors differing across modes. Inference in sample surveys generally proceeds by correcting for selectivity—for example, by applying calibration estimators—and ignoring measurement error. When a survey is conducted repeatedly, such inferences are valid only if the measurement error remains constant between surveys. In sequential mixed-mode surveys, it is likely that the mode composition of the overall response differs between subsequent editions of the survey, leading to variations in the total measurement error and invalidating classical inferences. An approach to inference in these circumstances, which is based on calibrating the mode composition of the respondents toward fixed levels, is proposed. Assumptions and risks are discussed and explored in a simulation and applied to the Dutch crime victimization survey.

[1]  Andreas Karlsson,et al.  Estimation in Surveys with Nonresponse , 2007, Technometrics.

[2]  F. Conrad,et al.  Spacing, Position, and Order Interpretive Heuristics for Visual Features of Survey Questions , 2004 .

[3]  R. Valliant,et al.  Investigating the Bias of Alternative Statistical Inference Methods in Sequential Mixed-Mode Surveys , 2012 .

[4]  P. Chisnall.  Mail and Internet Surveys: The Tailored Design Method , 2007, Journal of Advertising Research.

[5]  Jolene D. Smyth,et al.  The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys , 2007 .

[6]  Melanie C. Green,et al.  Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias , 2003 .

[7]  Jolene D Smyth,et al.  Visual Design, Order Effects, and Respondent Characteristics in a Self-Administered Survey , 2007 .

[8]  Paul Beatty,et al.  RESPONSE RATES AND RESPONSE CONTENT IN MAIL VERSUS FACE-TO-FACE SURVEYS , 1994 .

[9]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[10]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[11]  W. Aquilino.  INTERVIEW MODE EFFECTS IN SURVEYS OF DRUG AND ALCOHOL USE: A FIELD EXPERIMENT , 1994 .

[12]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[13]  Carl-Erik Särndal,et al.  Model Assisted Survey Sampling , 1997 .

[14]  Barry Schouten,et al.  Disentangling mode-specific selection and measurement bias in social surveys. , 2013, Social science research.

[15]  Willem E. Saris,et al.  Mixed mode designs: finding the balance between nonresponse bias and mode effects , 2005 .

[16]  Sixten Lundström,et al.  Estimation in Surveys with Nonresponse: Särndal/Estimation in Surveys with Nonresponse , 2005 .

[17]  Peter Lynn,et al.  Assessing the Effect of Data Collection Mode on Measurement , 2010 .

[18]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[19]  Thomas Lumley,et al.  Complex Surveys: A Guide to Analysis Using R , 2010 .

[20]  Vera Toepoel,et al.  Design of Web Questionnaires: The Effect of Layout in Rating Scales , 2006 .

[21]  W. Aquilino,et al.  EFFECTS OF INTERVIEW MODE ON SELF-REPORTED DRUG USE , 1990 .

[22]  Geert Loosveldt,et al.  Evaluating Relative Mode Effects in Mixed-Mode Surveys: , 2013 .

[23]  Paul Biemer,et al.  Nonresponse Bias and Measurement Bias in a Comparison of Face to Face and Telephone Interviewing , 2001 .

[24]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[25]  Don A. Dillman,et al.  Survey Mode as a Source of Instability in Responses across Surveys , 2005 .

[26]  Scott Fricker,et al.  An Experimental Comparison of Web and Telephone Surveys , 2005 .

[27]  Shirley Dex,et al.  On the experience and evidence about mixing modes of data collection in large-scale surveys where the web is used as one of the modes in data collection , 2011 .

[28]  F. Conrad,et al.  Color, Labels, and Interpretive Heuristics for Response Scales , 2007 .

[29]  Robert D. Tortora,et al.  Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet , 2009 .

[30]  E. Leeuw,et al.  To mix or not to mix data collection modes in surveys. , 2005 .

[31]  Bart Buelens,et al.  On the Necessity to Include Personal Interviewing in Mixed-Mode Surveys , 2010 .

[32]  Geert Molenberghs,et al.  A Method for Evaluating Mode Effects in Mixed-mode Surveys , 2010 .