Sample Size Planning for Detecting Mediation Effects: A Power Analysis Procedure Considering Uncertainty in Effect Size Estimates

Abstract When planning mediation studies, researchers are often interested in the sample size needed to achieve adequate power for testing mediation. Power depends on population effect sizes, which are unknown in practice. In conventional power analysis, effect size estimates, however, are often used as population values, which could result in underpowered studies. Uncertainty in effect size estimates has been considered in other sample size planning contexts (e.g., t-test, ANOVA), but has not been handled properly for planning mediation studies. In the current study, we proposed an easy-to-use sample size planning method for testing mediation with uncertainty in effect size estimates considered. We conducted simulation studies to demonstrate the impact of uncertainty in effect size estimates on power of testing mediation, and to provide sample size suggestions under different levels of uncertainty. Empirical examples were provided to illustrate the application of our method. R functions and a web application were developed to facilitate implementation.

[1]  A. Hayes.  Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach , 2013 .

[2]  James K. Harter,et al.  INTERPERSONAL RELATIONS AND GROUP PROCESSES Wealth and Happiness Across the World : Material Prosperity Predicts Life Evaluation , Whereas Psychosocial Prosperity Predicts Positive Feeling , 2010 .

[3]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[4]  E. O’Brien,et al.  Worth the Wait? Leisure Can Be Just as Enjoyable With Work Left Undone , 2017, Psychological science.

[5]  Joseph R. Rausch,et al.  Sample size planning for the standardized mean difference: accuracy in parameter estimation via narrow confidence intervals. , 2006, Psychological methods.

[6]  Ken Kelley,et al.  A Novel Measure of Effect Size for Mediation Analysis , 2017, Psychological methods.

[7]  Marc A. Brackett,et al.  Classroom emotional climate, student engagement, and academic achievement , 2012 .

[8]  Sample Size for Joint Testing of Indirect Effects , 2015, Prevention Science.

[9]  D. Spiegelhalter,et al.  Monitoring clinical trials: conditional or predictive power? , 1986, Controlled clinical trials.

[10]  M. A. Best.  Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[11]  Scott E. Maxwell,et al.  Designing Experiments and Analyzing Data: A Model Comparison Perspective , 1990 .

[12]  Douglas J. Taylor,et al.  BIAS IN LINEAR MODEL POWER AND SAMPLE SIZE CALCULATION DUE TO ESTIMATING NONCENTRALITY. , 1996, Communications in statistics: theory and methods.

[13]  D. Lindley.  The choice of sample size , 1997 .

[14]  Matthew S. Fritz,et al.  Distribution of the product confidence limits for the indirect effect: Program PRODCLIN , 2007, Behavior research methods.

[15]  David P Mackinnon,et al.  Explanation of Two Anomalous Results in Statistical Mediation Analysis , 2012, Multivariate behavioral research.

[16]  S. West,et al.  A comparison of methods to test mediation and other intervening variable effects. , 2002, Psychological methods.

[17]  Fang Liu,et al.  An Extension of Bayesian Expected Power and Its Application in Decision Making , 2010, Journal of biopharmaceutical statistics.

[18]  Mike W. L. Cheung,et al.  Comparison of methods for constructing confidence intervals of standardized indirect effects , 2009, Behavior research methods.

[19]  Kristopher J Preacher,et al.  Advances in mediation analysis: a survey and synthesis of new developments. , 2015, Annual review of psychology.

[20]  Zhiyong Zhang,et al.  Estimating and Testing Mediation Effects with Censored Data , 2011 .

[21]  Keith E. Muller,et al.  BIAS IN LINEAR MODEL POWER AND SAMPLE SIZE DUE TO ESTIMATING VARIANCE. , 1997, Communications in statistics: theory and methods.

[22]  R. Waldinger,et al.  The Long Reach of Nurturing Family Environments , 2016, Psychological science.

[23]  Zhiyong Zhang,et al.  Monte Carlo based statistical power analysis for mediation models: methods and software , 2013, Behavior Research Methods.

[24]  S. Maxwell,et al.  Ethics and Sample Size Planning , 2011 .

[25]  D. Joanes,et al.  Comparing measures of sample skewness and kurtosis , 1998 .

[26]  Kai Wang.  Understanding Power Anomalies in Mediation Analysis , 2018, Psychometrika.

[27]  R. Hoyle.  Statistical Strategies for Small Sample Research , 1999 .

[28]  Kristopher J Preacher,et al.  Moderated Mediation Analysis Using Bayesian Methods , 2015 .

[29]  Ying Yuan,et al.  Sample size determination for mediation analysis of longitudinal data , 2018, BMC Medical Research Methodology.

[30]  P. Shrout,et al.  Mediation in experimental and nonexperimental studies: new procedures and recommendations. , 2002, Psychological methods.

[31]  Stephen D. Short,et al.  Determining Power and Sample Size for Simple and Complex Mediation Models , 2017 .

[32]  D. A. Kenny,et al.  The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. , 1986, Journal of personality and social psychology.

[33]  D. Culler,et al.  Comparison of methods , 2000 .

[34]  Ken Kelley,et al.  Sample Size Planning , 2010, The SAGE Encyclopedia of Research Design.

[35]  Han Du,et al.  A Bayesian Power Analysis Procedure Considering Uncertainty in Effect Size Estimates from a Meta-analysis , 2016, Multivariate behavioral research.

[36]  Kristopher J Preacher,et al.  Mediation Models for Longitudinal Data in Developmental Research , 2009 .

[37]  Joseph R. Rausch,et al.  Sample size planning for statistical power and accuracy in parameter estimation. , 2008, Annual review of psychology.

[38]  D. Mackinnon,et al.  A Note on Testing Mediated Effects in Structural Equation Models , 2016, Educational and psychological measurement.

[39]  Xitao Fan,et al.  Monotonicity of effect sizes: Questioning kappa-squared as mediation effect size measure. , 2015, Psychological methods.

[40]  Marco Perugini,et al.  Safeguard Power as a Protection Against Imprecise Power Estimates , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[41]  Matthew S. Fritz,et al.  PSYCHOLOGICAL SCIENCE Research Article Required Sample Size to Detect the Mediated Effect , 2022 .

[42]  David P. MacKinnon,et al.  Mediation analysis with longitudinal data. , 2012 .

[43]  Ken Kelley,et al.  Accuracy in parameter estimation for a general class of effect sizes: A sequential approach. , 2017, Psychological methods.

[44]  Scott M. Hofer,et al.  Longitudinal data analysis : a practical guide for researchers in aging, health, and social sciences , 2013 .

[45]  D. Mackinnon.  Introduction to Statistical Mediation Analysis , 2008 .

[46]  M. Wyllie,et al.  Is it worth the wait? , 2009, BJU international.

[47]  S. Maxwell,et al.  Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? , 2015, The American psychologist.

[48]  Ying Yuan,et al.  Bayesian mediation analysis. , 2009, Psychological methods.

[49]  H. Kraemer.  A mediator effect size in randomized clinical trials , 2014, International journal of methods in psychiatric research.

[50]  Ken Kelley,et al.  Sample size for multiple regression: obtaining regression coefficients that are accurate, not simply significant. , 2003, Psychological methods.

[51]  Fulvio De Santis,et al.  Using historical data for Bayesian sample size determination , 2007 .

[52]  M. Sobel.  Asymptotic Confidence Intervals for Indirect Effects in Structural Equation Models , 1982 .

[53]  Kristopher J Preacher,et al.  SPSS and SAS procedures for estimating indirect effects in simple mediation models , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[54]  D. A. Kenny,et al.  Power Anomalies in Testing Mediation , 2014, Psychological science.

[55]  Ulf Böckenholt,et al.  Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach. , 2016, Psychological methods.

[56]  David P Mackinnon,et al.  Confidence Limits for the Indirect Effect: Distribution of the Product and Resampling Methods , 2004, Multivariate behavioral research.

[57]  Matthew S. Fritz,et al.  Mediation analysis. , 2019, Annual review of psychology.

[58]  Samantha F. Anderson,et al.  Addressing the “Replication Crisis”: Using Original Studies to Design Replication Studies with Appropriate Statistical Power , 2017, Multivariate behavioral research.

[59]  Ken Kelley,et al.  Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty , 2017, Psychological science.

[60]  David P. MacKinnon,et al.  R2 effect-size measures for mediation analysis , 2009, Behavior research methods.

[61]  P. Mielke,et al.  Least Sum of Euclidean Regression Residuals: Estimation of Effect Size , 2002, Psychological reports.

[62]  P. de Heus.  R squared effect-size measures and overlap between direct and indirect effect in mediation analysis , 2011, Behavior Research Methods.

[63]  Kristopher J Preacher,et al.  Effect size measures for mediation models: quantitative strategies for communicating indirect effects. , 2011, Psychological methods.

[64]  Kristopher J Preacher,et al.  Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models , 2008, Behavior research methods.