Mode effect or question wording? Measurement error in mixed mode surveys

Members of a high quality, probability-based Internet panel were randomly assigned to one of two modes: (1) computer assisted telephone interview or (2) web survey. Within each mode the same series of split ballot experiments on question format were conducted. We tested the effect of unfolding opinion questions in multiple steps vs. asking a complete question in one-step, and the effect of fully verbal labelling versus end-point labelling of response categories within and between the two modes. We found small direct mode effects, but no interaction. Unfolding (two-step question) had a larger effect. When using a mixed-mode design it is advisable to avoid unfolding formats in the telephone interview and use the complete (one step) question format in both modes. Full labelling is mildly preferred above labelling endpoints only. The absence of an interaction effect is an encouraging result for mixed-mode surveys.

[1]  Donald D. Rugg,et al.  EXPERIMENTS IN WORDING QUESTIONS: II , 1941 .

[2]  Robert M. Groves,et al.  Actors and Questions in Telephone and Personal Interview Surveys , 1979 .

[3]  Norbert Schwarz,et al.  Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments , 1985 .

[4]  Norbert Schwarz,et al.  Response Effects in Surveys , 1987 .

[5]  李幼升,et al.  Ph , 1989 .

[6]  F. Strack,et al.  The impact of administration mode on response effects in survey measurement , 1991 .

[7]  D. Leeuw,et al.  Data Quality in Mail, Telephone and Face to Face Surveys. , 1992 .

[8]  Jon A. Krosnick,et al.  Designing Rating Scales for Effective Measurement in Surveys , 1997 .

[9]  Eric R. Ziegel,et al.  Survey measurement and process quality , 1998 .

[10]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[11]  E. Leeuw,et al.  To mix or not to mix data collection modes in surveys. , 2005 .

[12]  Don A. Dillman,et al.  Survey Mode as a Source of Instability in Responses across Surveys , 2005 .

[13]  Michael W. Link,et al.  Effects of survey mode on self-reports of adult alcohol consumption: a comparison of mail, web and telephone approaches. , 2005, Journal of studies on alcohol.

[14]  Jolene D. Smyth,et al.  The Effects of Mode and Format on Answers to Scalar Questions in Telephone and Web Surveys , 2007 .

[15]  P. Chisnall.  Mail and Internet Surveys: The Tailored Design Method , 2007, Journal of Advertising Research.

[16]  Edith D. de Leeuw,et al.  Advances in Telephone Survey Methodology , 2007 .

[17]  F. Kreuter,et al.  Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity , 2008 .

[18]  Bill Blyth,et al.  Mixed mode: The Only ‘Fitness’ Regime? , 2008 .

[19]  Mick P. Couper,et al.  Designing Effective Web Surveys: Preface , 2008 .

[20]  Peter Lynn,et al.  Assessing the Effect of Data Collection Mode on Measurement , 2010 .