Where Am I? A Meta-Analysis of Experiments on the Effects of Progress Indicators for Web Surveys

The use of progress indicators seems to be standard in many online surveys. Researchers include them in surveys in the hope they will help reduce drop-off rates. However, there is no consensus in the literature regarding their effects. In this meta-analysis, we analyzed 32 randomized experiments comparing drop-off rates of an experimental group who completed an online survey in which a progress indicator was shown to drop-off rates of a control group to whom the progress indicator was not shown. In all the studies, a drop-off was defined as a discontinuance of the survey (at any point) after it has begun, resulting in failure to complete the survey. Three types of progress indicators were analyzed: constant, fast-to-slow, and slow-to-fast. Our results show that, overall, using a constant progress indicator does not significantly help reduce drop-offs and that effectiveness of the progress indicator varies depending on the speed of indicator: Fast-to-slow indicators reduced drop-offs, whereas slow-to-fast indicators increased drop-offs. We also found that among the studies in which a small incentive was promised, showing a constant progress indicator increased the drop-off rate. These findings question the common belief that progress indicators help reduce drop-off rates.

[1]  Smith Ma.  It's what's up-front that counts! , 1989 .

[2]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[3]  Angela P. Wetzel.  Internet, mail, and mixed‐mode surveys: The tailored design method , 2010 .

[4]  Alex Paul Conn,et al.  Time affordances: the time factor in diagnostic usability heuristics , 1995, CHI '95.

[5]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[6]  Robert Bell,et al.  Rethinking the progress bar , 2007, UIST.

[7]  Leif D. Nelson,et al.  You’re Having Fun When Time Flies , 2010, Psychological science.

[8]  Brad A. Myers.  INCENSE: A system for displaying data structures , 1983, SIGGRAPH.

[9]  M. Couper,et al.  Web Surveys , 2001 .

[10]  Brad A. Myers,et al.  The importance of percent-done progress indicators for computer-human interfaces , 1985, CHI '85.

[11]  Andreas Rosenblad,et al.  Introduction to Meta‐Analysis by Michael Borenstein, Larry V. Hedges, Julian P.T. Higgins, Hannah R. Rothstein , 2009 .

[12]  Roger Tourangeau,et al.  Impact of progress feedback on task completion: first impressions matter , 2005, CHI EA '05.

[13]  Kay Dickersin,et al.  Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm , 2006 .

[14]  Byron C. Wallace,et al.  Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data , 2009, BMC medical research methodology.

[15]  Mario Köppen,et al.  Color Effect on Subjective Perception of Progress Bar Speed , 2011, 2011 Third International Conference on Intelligent Networking and Collaborative Systems.

[16]  Roger Tourangeau,et al.  Should I Stay or Should I go: The Effects of Progress Feedback, Promised Task Duration, and Length of Questionnaire on Completing Web Surveys , 2011 .

[17]  Karel G M Moons,et al.  More than numbers: the power of graphs in meta-analysis. , 2008, American journal of epidemiology.

[18]  S. Crawford,et al.  The Impact of Textual Messages of Encouragement on Web Survey Breakoffs: An Experiment , 2009 .

[19]  M. Traugott,et al.  Web survey design and administration. , 2001, Public opinion quarterly.

[20]  Scott B. Morris.  Book Review: Hunter, J. E., & Schmidt, F. L. (2004). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings (2nd ed.). Thousand Oaks, CA: Sage , 2008 .

[21]  Roger Tourangeau,et al.  The impact of progress indicators on task completion , 2010, Interact. Comput..

[22]  R. Block,et al.  Prospective and retrospective duration judgments: an executive-control perspective. , 2004, Acta neurobiologiae experimentalis.

[23]  James R. Lewis,et al.  Effect of Auditory Waiting Cues on Time Estimation in Speech Recognition Telephony Applications , 2002, Int. J. Hum. Comput. Interact..

[24]  Hannah R Rothstein,et al.  A basic introduction to fixed‐effect and random‐effects models for meta‐analysis , 2010, Research synthesis methods.

[25]  Geert Loosveldt,et al.  An experimental study on the effects of personalization, survey length statements, progress indicators, and survey sponsor logos in web surveys , 2006 .

[26]  Lars Kaczmirek,et al.  Human-Survey Interaction : Usability and Nonresponse in Online Surveys , 2008 .

[27]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[28]  Lois A. Ritter,et al.  Conducting Online Surveys , 2011 .

[29]  R. Block,et al.  Prospective and retrospective duration judgments: A meta-analytic review , 1997, Psychonomic bulletin & review.

[30]  Douglas G. Altman,et al.  Chapter 9: Analysing Data and Undertaking Meta-Analyses , 2008 .

[31]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[32]  Harukazu Tsuruta,et al.  Development and validation of MIX: comprehensive free software for meta-analysis of causal research data , 2006, BMC medical research methodology.

[33]  Uwe Matzat,et al.  Effects of Different Types of Progress Indicators on Drop-Out Rates in Web Surveys , 2009 .

[34]  Ilana Ritov,et al.  Routine and the perception of time. , 2003, Journal of experimental psychology. General.

[35]  S. Bown,et al.  A light at the end of the tunnel , 1998, Gut.

[36]  R F Galbraith,et al.  A note on graphical presentation of estimated odds ratios from several clinical trials. , 1988, Statistics in medicine.

[37]  Alison Moore,et al.  'A light at the end of the tunnel'. , 2015, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[38]  Karel GM Moons,et al.  A systematic comparison of software dedicated to meta-analysis of causal studies , 2007, BMC medical research methodology.

[39]  Ron Jacobs.  A light at the end of the tunnel... : progress indicators and their influence on completion rates, per-page dropout rates and task durations in online browser surveys , 2007 .

[40]  Juan Carlos Espinosa,et al.  Comprehensive Meta-Analysis , 2004 .

[41]  Terry Macpherson,et al.  An Empirical Evaluation of Three Web Survey Design Principles , 2005 .

[42]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[43]  F. Al-Shamali,et al.  Author Biographies. , 2015, Journal of social work in disability & rehabilitation.

[44]  Jonathon D. Crystal,et al.  The psychology of time , 2007, Behavioural Processes.