A Sequential Mixed-Mode Experiment in the U.S. National Monitoring the Future Study

Abstract The national Monitoring the Future (MTF) study examines substance use among adolescents and adults in the United States and has used paper questionnaires since it began in 1975. The current experiment tested three conditions as compared to the standard MTF follow-up protocol (i.e., MTF Control) for the first MTF follow-up survey at ages 19/20 years (i.e., one or two years after high school graduation). The MTF Control group included participants who completed in-school baseline surveys in the 12th grade in 2012–2013 and who were selected to participate in the first follow-up survey in 2014 (n = 2,451). A supplementary sample of participants who completed the 12th grade baseline survey in 2012 or 2013 but were not selected to participate in the main MTF follow-up (n = 4,950) were recruited and randomly assigned to one of three experimental conditions: (1) Mail Push, (2) Web Push, (3) Web Push + E-mail. Results indicated that the overall response rate was lower in Condition 2 compared to MTF Control and to Condition 1; there were no differences between Condition 3 and other conditions. Web response was highest in Condition 3; among web responders, smartphone response was also highest in Condition 3. Subgroup differences also emerged such that, for example, compared to white participants, Hispanics had greater odds of web (versus paper) response and blacks had greater odds of smartphone (versus computer or tablet) response. Item nonresponse was lowest in the Web Push conditions (compared to MTF Control) and on the web survey (compared to paper). Compared to MTF Control, Condition 3 respondents reported higher rates of alcohol use in the past 30 days. The total cost was lowest for Condition 3. Overall, the Condition 3 Web Push + E-mail design is promising. Future research is needed to continue to examine the implications of web and mobile response in large, national surveys.

[1]  Jolene D. Smyth,et al.  Using the Internet to Survey Small Towns and Communities: Limitations and Possibilities in the Early 21st Century , 2010 .

[2]  L. Johnston,et al.  Monitoring the Future national survey results on drug use, 1975-2017: Volume I, secondary school students , 2005 .

[3]  Multi-Mode and Method Experiment in a Study of Nurses , 2010 .

[4]  Anders Holmberg,et al.  Contact strategies to improve participation via the web in a mixed-mode mail and web survey , 2010 .

[5]  B. Schwartz.  The Paradox of Choice: Why More Is Less , 2004 .

[6]  Alexandru Cernat,et al.  The role of email addresses and email contact in encouraging web response in a mixed mode design , 2014 .

[7]  Jennifer Taylor,et al.  Evaluation of the innovations implemented in the 2009 Canadian Census Test , 2010 .

[8]  Bengü Börkan,et al.  The Mode Effect in Mixed-Mode Surveys , 2010 .

[9]  Glenn D. Israel,et al.  Combining Mail and E-Mail Contacts to Facilitate Participation in Mixed-Mode Surveys , 2013 .

[10]  Mick P. Couper,et al.  MEASURING SURVEY QUALITY IN A CASIC ENVIRONMENT , 2002 .

[11]  Roger Tourangeau,et al.  The Science of Web Surveys , 2013 .

[12]  Lars Kaczmirek,et al.  The Mode of Invitation for Web Surveys , 2012 .

[13]  Michael J. Stern,et al.  The Use of Client-side Paradata in Analyzing the Effects of Visual Layout on Changing Responses in Web Surveys , 2008 .

[14]  Mick Couper,et al.  The Use of Paradata in Survey Research , 2022 .

[15]  S. McCabe.  Comparison of Web and Mail Surveys in Collecting Illicit Drug Use Data: A Randomized Experiment , 2004, Journal of drug education.

[16]  Edith D. de Leeuw,et al.  Total survey error in practice , 2017 .

[17]  Don A. Dillman,et al.  Surveying the General Public over the Internet Using Address-Based Sampling and Mail Contact Procedures , 2011 .

[18]  Dirk Heerwegh,et al.  Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey , 2003 .

[19]  Glenn D. Israel,et al.  Obtaining Responses by Mail or Web: Response Rates and Data Consequences , 2009 .

[20]  Martyn Denscombe,et al.  Web-Based Questionnaires and the Mode Effect , 2006 .

[21]  Morgan M. Millar,et al.  Improving Response To Web and Mixed-Mode Surveys , 2011 .

[22]  Mick P Couper,et al.  Mode effects for collecting alcohol and other drug use data: Web and U.S. mail. , 2002, Journal of studies on alcohol.

[23]  Michael W. Link,et al.  Does the Timing of Offering Multiple Modes of Return Hurt the Response Rate , 2010 .

[24]  Lars Kaczmirek,et al.  The Effectiveness of Mailed Invitations for Web Surveys and the Representativeness of Mixed-Mode versus Internet-only Samples , 2014 .

[25]  Jolene D. Smyth,et al.  Does Giving People Their Preferred Survey Mode Actually Increase Survey Participation Rates? An Experimental Examination , 2012 .

[26]  L. Johnston,et al.  The Monitoring the Future Project after Four Decades: Design and Procedures. Monitoring the Future Occasional Paper Series. Paper 82. , 2015 .

[27]  Rebecca L. Medway,et al.  When More Gets You Less: A Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates , 2012 .

[28]  K. Manfreda,et al.  Web Surveys versus other Survey Modes: A Meta-Analysis Comparing Response Rates , 2008 .

[29]  Megan E. Patrick,et al.  Monitoring the Future national survey results on drug use, 1975-2017: Overview, key findings on adolescent drug use , 2016 .