The Mode Effect in Mixed-Mode Surveys

Web surveys can suffer from their nonrandom nature (coverage error) and low response rate (nonresponse error). Therefore, web surveys should be supported by mail survey to eliminate these problems. However, using different survey methods together may introduce another problem: the mode effect. This experimental study investigated the mode effect between two survey modes. A randomly selected group of 1,500 teachers were assigned to two experimental groups, one of which received mail surveys, while the other received web surveys. Nonrespondents in both groups were followed up with the opposite mode. Overall, results show that there is no mode effect between mail surveys and web surveys on psychometric quality of the rating scales and data quality (item nonresponse rate) of the survey except regarding respondents’ age and unit-response rate. Our findings indicate that web surveys had a substantially lower unit-response rate than mail surveys and that web survey respondents are significantly younger than mail survey respondents.

[1]  Thomas G. Reed.  Elementary principal emotional intelligence, leadership behavior, and openness: An exploratory study , 2005 .

[2]  R. Goddard.  A Theoretical and Empirical Analysis of the Measurement of Collective Efficacy: The Development of a Short Form , 2002 .

[3]  N. Allen,et al.  Computerized and Written Questionnaires: Are They Equivalent? , 1994 .

[4]  Martyn Denscombe,et al.  Web-Based Questionnaires and the Mode Effect , 2006 .

[5]  M. Elliott,et al.  Conducting Research Surveys Via E-Mail and The Web , 2001 .

[6]  Philip Hans Franses,et al.  Confidence Intervals for Cronbach's Coefficient Alpha Values , 2003 .

[7]  P. Chisnall.  Mail and Internet Surveys: The Tailored Design Method , 2007, Journal of Advertising Research.

[8]  Robert D. Tortora,et al.  Principles for Constructing Web Surveys , 1998 .

[9]  Russel L. Thompson,et al.  A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys , 2000 .

[10]  Alan I. Wilson,et al.  Internet based marketing research: a serious alternative to traditional research methods? , 2003 .

[11]  Earl R. Babbie,et al.  The practice of social research , 1969 .

[12]  Judy Eaton,et al.  Using the Internet for Organizational Research: A Study of Cynicism in the Workplace , 2002, Cyberpsychology Behav. Soc. Netw..

[13]  F. Langerak,et al.  Non-Probability Sampling for WWW Surveys: A Comparison of Methods , 1998 .

[14]  Heath McDonald.  A comparison of online and postal data collection methods in marketing research , 2012 .

[15]  Bernard J. Jansen,et al.  E-survey methodology , 2007 .

[16]  Don A. Dillman,et al.  INFLUENCE OF AN INVITATION TO ANSWER BY TELEPHONE ON RESPONSE TO CENSUS QUESTIONNAIRES , 1994 .

[17]  Matthias Schonlau,et al.  Web Surveys as Part of a Mixed-Mode Strategy for Populations that cannot be Contacted by E-Mail , 2003 .

[18]  M. Singh,et al.  Electronic Data Collection Methods , 2007 .

[19]  M. Birnbaum.  Human research and data collection via the internet. , 2004, Annual review of psychology.

[20]  Robert D. Tortora,et al.  Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet , 2009 .

[21]  Norman M. Bradburn,et al.  Chapter 8 – Response Effects1 , 1983 .

[22]  Samuel Garrett-Jones,et al.  How Demographic Characteristics Affect Mode Preference in a Postal/Web Mixed-Mode Survey of Australian Researchers , 2007 .

[23]  F. Strack,et al.  The impact of administration mode on response effects in survey measurement , 1991 .

[24]  D. Leeuw,et al.  Data Quality in Mail, Telephone and Face to Face Surveys. , 1992 .

[25]  Martyn Denscombe,et al.  The Length of Responses to Open-Ended Questions , 2008 .

[26]  Sunghee Lee.  An Evaluation of Nonresponse and Coverage Errors in a Prerecruited Probability Web Panel Survey , 2006 .

[27]  W. Hoy,et al.  Academic Emphasis of Urban Elementary Schools and Student Achievement in Reading and Mathematics: A Multilevel Analysis , 2000 .

[28]  Don A. Dillman,et al.  Questionnaire Context as a Source of Response Differences in Mail and Telephone Surveys , 1992 .

[29]  Barbara M. Wildemuth,et al.  Pixels or pencils? The relative effectiveness of Web-based versus paper surveys , 2004 .

[30]  R. Dixon.  Electronic vs. Conventional Surveys , 2007 .

[31]  Michael E. Whitcomb,et al.  Mixed-Mode Contacts In Web Surveys Paper is Not Necessarily Better , 2007 .

[32]  J. Epstein,et al.  Insuring sample equivalence across internet and paper-and-pencil assessments , 2001, Comput. Hum. Behav..

[33]  Samuel J. Best,et al.  An Assessment of the Generalizability of Internet Surveys , 2001 .