Small sample mediation testing: misplaced confidence in bootstrapped confidence intervals.

Bootstrapping is an analytical tool commonly used in psychology to test the statistical significance of the indirect effect in mediation models. Bootstrapping proponents have particularly advocated for its use for samples of 20-80 cases. This advocacy has been heeded, especially in the Journal of Applied Psychology, as researchers are increasingly utilizing bootstrapping to test mediation with samples in this range. We discuss reasons to be concerned with this escalation, and in a simulation study focused specifically on this range of sample sizes, we demonstrate not only that bootstrapping has insufficient statistical power to provide a rigorous hypothesis test in most conditions but also that bootstrapping has a tendency to exhibit an inflated Type I error rate. We then extend our simulations to investigate an alternative empirical resampling method as well as a Bayesian approach and demonstrate that they exhibit comparable statistical power to bootstrapping in small samples without the associated inflated Type I error. Implications for researchers testing mediation hypotheses in small samples are presented. For researchers wishing to use these methods in their own research, we have provided R syntax in the online supplemental materials.

[1]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[2]  David P Mackinnon,et al.  Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods , 2004, Multivariate behavioral research.

[3]  Lawrence R. James,et al.  A Tale of Two Methods , 2006 .

[4]  Clement A. Stone,et al.  The robustness of estimates of total indirect effects in covariance structure models estimated by maximum , 1990 .

[5]  P. Shrout,et al.  Mediation in experimental and nonexperimental studies: new procedures and recommendations. , 2002, Psychological methods.

[6]  Christopher W. Bauman,et al.  The pursuit of missing information in negotiation , 2012 .

[7]  Kristopher J Preacher,et al.  SPSS and SAS procedures for estimating indirect effects in simple mediation models , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[8]  John K. Kruschke,et al.  The Time Has Come : Bayesian Methods for Data Analysis in the Organizational Sciences , 2012 .

[9]  Anthony J Bishara,et al.  Testing the significance of a correlation with nonnormal data: comparison of Pearson, Spearman, transformation, and resampling approaches. , 2012, Psychological methods.

[10]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[11]  E. Ziegel.  Permutation, Parametric, and Bootstrap Tests of Hypotheses (3rd ed.) , 2005 .

[12]  Winny Shen,et al.  Samples in applied psychology: over a decade of research in review. , 2011, The Journal of applied psychology.

[13]  Laura M. Stapleton,et al.  A Comparison of Single Sample and Bootstrap Methods to Assess Mediation in Cluster Randomized Trials , 2006, Multivariate behavioral research.

[14]  Dave Bartram,et al.  Choosing the best method for local validity estimation: relative accuracy of meta-analysis versus a local study versus Bayes-analysis. , 2007, The Journal of applied psychology.

[15]  Michael A. Kepler,et al.  Using SAS to conduct nonparametric residual bootstrap multilevel modeling with a small number of groups , 2006, Comput. Methods Programs Biomed..

[16]  John E. Hunter,et al.  Task Differences as Moderators of Aptitude Test Validity in Selection: A Red Herring , 1981 .

[17]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[18]  David P MacKinnon,et al.  Four applications of permutation methods to testing a single-mediator model , 2012, Behavior research methods.

[19]  Patrick J. Rosopa,et al.  Design Characteristics The Relative Validity of Inferences About Mediation as a Function of Research , 2008 .

[20]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[21]  Charles E. Lance,et al.  More Statistical and Methodological Myths and Urban Legends , 2011 .

[22]  A. Brix.  Bayesian Data Analysis, 2nd edn , 2005 .

[23]  Ying Yuan,et al.  Bayesian mediation analysis. , 2009, Psychological methods.

[24]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[25]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[26]  Jeffrey R. Edwards,et al.  To prosper, organizational psychology should … overcome methodological barriers to progress , 2008 .

[27]  Caesar Saloma,et al.  Things I have learned so far , 2008 .

[28]  P. Shrout,et al.  Mediation in experimental and nonexperimental studies: new procedures and recommendations. , 2002, Psychological methods.

[29]  Kristopher J Preacher,et al.  A general multilevel SEM framework for assessing multilevel mediation. , 2010, Psychological methods.

[30]  M. Sobel.  Asymptotic Confidence Intervals for Indirect Effects in Structural Equation Models , 1982 .

[31]  Walter Krämer,et al.  Review of Modern applied statistics with S, 4th ed. by W.N. Venables and B.D. Ripley. Springer-Verlag 2002 , 2003 .

[32]  M. Zyphur,et al.  Bayesian Estimation and Inference , 2015 .

[33]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[34]  Gordon W. Cheung,et al.  Testing Mediation and Suppression Effects of Latent Variables , 2008 .

[35]  L. J. Williams,et al.  Recent Advances in Causal Modeling Methods for Organizational and Management Research , 2003 .

[36]  David P Mackinnon,et al.  Explanation of Two Anomalous Results in Statistical Mediation Analysis , 2012, Multivariate behavioral research.

[37]  Andrew D. Martin,et al.  MCMCpack: Markov chain Monte Carlo in R , 2011 .

[38]  Brian D. Ripley,et al.  Modern applied statistics with S, 4th Edition , 2002, Statistics and computing.

[39]  Kristopher J Preacher,et al.  Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models , 2008, Behavior research methods.

[40]  L. James,et al.  Mediators, Moderators, and Tests for Mediation. , 1984 .

[41]  B. Efron.  The Bootstrap and Modern Statistics , 2000 .

[42]  Victoria Savalei,et al.  Assessing Mediational Models: Testing and Interval Estimation for Indirect Effects , 2010, Multivariate behavioral research.

[43]  D. Mackinnon,et al.  Tests of the Three-Path Mediated Effect , 2008 .

[44]  Junqi Shi,et al.  Can't Get It Out of My Mind: Employee Rumination after Customer Mistreatment and Negative Mood in the Next Morning , 2013, The Journal of applied psychology.

[45]  Laura M. Stapleton,et al.  The Performance of Methods to Test Upper-Level Mediation in the Presence of Nonnormal Data , 2008, Multivariate behavioral research.

[46]  James O. Berger,et al.  The interplay of Bayesian and frequentist analysis , 2004 .

[47]  Michael J. Mannor,et al.  Statistical power and parameter stability when subjects are few and tests are many: comment on Peterson, Smith, Martorana, and Owens (2003). , 2006, The Journal of applied psychology.

[48]  James G. Combs.  Big Samples and Small Effects: Let's Not Trade Relevance and Rigor for Power , 2010 .

[49]  Jeffery A. LePine,et al.  Trust in Typical and High-Reliability Contexts: Building and Reacting to Trust among Firefighters , 2011 .