Power and Sample Size Determination for Multilevel Mediation in Three-Level Cluster-Randomized Trials

Mediation analyses supply a principal lens to probe the pathways through which a treatment acts upon an outcome because they can dismantle and test the core components of treatments and test how these components function as a coordinated system or theory of action. Experimental evaluation of mediation effects in addition to total effects has become increasingly common but literature has developed only limited guidance on how to plan mediation studies with multi-tiered hierarchical or clustered structures. In this study, we provide methods for computing the power to detect mediation effects in three-level cluster-randomized designs that examine individual- (level one), intermediate- (level two) or cluster-level (level three) mediators. We assess the methods using a simulation and provide examples of a three-level clinic-randomized study (individuals nested within therapists nested within clinics) probing an individual-, intermediate- or cluster-level mediator using the R package PowerUpR and its Shiny application.

[1]  Kristopher J Preacher,et al.  Advantages of Monte Carlo Confidence Intervals for Indirect Effects , 2012 .

[2]  David C. Atkins,et al.  Intraclass Correlation Associated with Therapists: Estimates and Applications in Planning Psychotherapy Research , 2011, Cognitive behaviour therapy.

[3]  A. Kozina.  Can the “My FRIENDS” Anxiety Prevention Programme Also be Used to Prevent Aggression? A Six-Month Follow-Up in a School , 2018 .

[4]  D. Kivlahan,et al.  Beliefs about evidence-based practices in addiction treatment: a survey of Veterans Administration program leaders. , 2004, Journal of substance abuse treatment.

[5]  Thomas D. Cook,et al.  Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation , 2015, Prevention Science.

[6]  Tyler J. VanderWeele,et al.  Conceptual issues concerning mediation, interventions and composition , 2009 .

[7]  Craig K. Enders,et al.  Centering Predictor Variables in Three-Level Contextual Models , 2017, Multivariate behavioral research.

[8]  Daniel L. Murphy,et al.  Three-Level Models for Indirect Effects in School- and Class-Randomized Experiments in Education , 2009 .

[9]  Amanda T. Woodward,et al.  Characteristics of Users of Consumer-Run Drop-In Centers Versus Clubhouses , 2009, The Journal of Behavioral Health Services & Research.

[10]  Ben Kelcey,et al.  Learning about Teachers' Literacy Instruction from Classroom Observations. , 2013 .

[11]  R. Schwartz.  Motivational interviewing (patient-centered counseling) to address childhood obesity. , 2010, Pediatric annals.

[12]  Craig K. Enders,et al.  Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. , 2007, Psychological methods.

[13]  Ben Kelcey,et al.  Teacher mathematical knowledge, instructional quality, and student outcomes: a multilevel quantile mediation analysis , 2019, School Effectiveness and School Improvement.

[14]  S. Kozlowski,et al.  Multilevel Theory, Research, a n d M e t h o d s i n Organizations Foundations, Extensions, and New Directions , 2022 .

[15]  Carl D. Westine,et al.  An Empirical Investigation of Variance Design Parameters for Planning Cluster-Randomized Trials of Science Achievement , 2013, Evaluation review.

[16]  Ci-Rong Li,et al.  Authentic leadership and employee creativity: testing the multilevel mediation model , 2017 .

[17]  S. Sterba.  Partially nested designs in psychotherapy trials: A review of modeling developments , 2017, Psychotherapy research : journal of the Society for Psychotherapy Research.

[18]  A. Kozina.  School-based prevention of anxiety using the "My FRIENDS" emotional resilience program: Six-month follow-up. , 2018, International journal of psychology : Journal international de psychologie.

[19]  Kimberly Hoagwood,et al.  The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study , 2012, Implementation Science.

[20]  T. Beasley,et al.  Tests of Mediation: Paradoxical Decline in Statistical Power as a Function of Mediator Collinearity , 2014, Journal of experimental education.

[21]  Hannah R. Rothstein,et al.  Synthesizing results of multiple randomized experiments to establish causality in mediation testing , 2015 .

[22]  Kristopher J Preacher,et al.  Testing Multilevel Mediation Using Hierarchical Linear Models , 2008 .

[23]  Ben Kelcey,et al.  Considerations for Designing Group Randomized Trials of Professional Development With Teacher Knowledge Outcomes , 2013 .

[24]  S. West,et al.  A comparison of methods to test mediation and other intervening variable effects. , 2002, Psychological methods.

[25]  Catherine P. Bradshaw,et al.  Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes , 2016, Evaluation review.

[26]  Jessaca K. Spybrook,et al.  Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators , 2017, Multivariate behavioral research.

[27]  D. Mackinnon,et al.  The Distribution of the Product Explains Normal Theory Mediation Confidence Interval Estimation , 2014, Multivariate behavioral research.

[28]  T J VanderWeele,et al.  Direct and Indirect Effects for Neighborhood-Based Clustered and Longitudinal Data , 2010, Sociological methods & research.

[29]  Jessaca K. Spybrook,et al.  Power for Detecting Treatment by Moderator Effects in Two- and Three-Level Cluster Randomized Trials , 2016 .

[30]  Carl D. Westine.  Finding Efficiency in the Design of Large Multisite Evaluations , 2016 .

[31]  I G Kreft,et al.  The Effect of Different Forms of Centering in Hierarchical Linear Models. , 1995, Multivariate behavioral research.

[32]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[33]  Jessaca Spybrook,et al.  Statistical Power for Causally Defined Indirect Effects in Group-Randomized Trials With Individual-Level Mediators , 2017 .

[34]  Laura M. Stapleton,et al.  A Comparison of Single Sample and Bootstrap Methods to Assess Mediation in Cluster Randomized Trials , 2006, Multivariate behavioral research.

[35]  A. Kazdin.  Mediators and mechanisms of change in psychotherapy research. , 2007, Annual review of clinical psychology.

[36]  S. Curenton,et al.  Does aggregate school-wide achievement mediate fifth grade outcomes for former early childhood education participants? , 2015, Developmental psychology.

[37]  D. Ford,et al.  Beliefs and Attitudes Associated With the Intention to Not Accept the Diagnosis of Depression Among Young Adults , 2005, The Annals of Family Medicine.

[38]  Arend M. Kuyper,et al.  The Variance of Intraclass Correlations in Three- and Four-Level Models , 2012 .

[39]  Kathleen J. Roth,et al.  Comparing the Effects of Analysis-of-Practice and Content-Based Professional Development on Teacher and Student Outcomes in Science , 2018, American Educational Research Journal.

[40]  David Evans,et al.  Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. , 2003, Journal of clinical nursing.

[41]  Jill Manit,et al.  National Survey on Drug Use and Health , 2009 .

[42]  Andrew R Willan,et al.  Randomizing patients by family practice: sample size estimation, intracluster correlation and data analysis. , 2003, Family practice.

[43]  T. Freeman,et al.  Brief interventions: good in theory but weak in practice. , 2004, Drug and alcohol review.

[44]  L. Hedges,et al.  Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education , 2013, Evaluation review.

[45]  Jeffrey R. Edwards,et al.  Improving Our Understanding of Moderation and Mediation in Strategic Management Research , 2017 .

[46]  Jeroen K. Vermunt,et al.  A multi‐level mediation model of the relationships between team autonomy, individual task design and psychological well‐being , 2007 .

[47]  Valerie B. Shapiro,et al.  Prevention System Mediation of Communities That Care Effects on Youth Outcomes , 2014, Prevention Science.

[48]  Michael Scharkow,et al.  The Relative Trustworthiness of Inferential Tests of the Indirect Effect in Statistical Mediation Analysis , 2013, Psychological science.

[49]  D. A. Kenny,et al.  Power Anomalies in Testing Mediation , 2014, Psychological science.

[50]  L. Dickinson,et al.  Motivational interviewing improves depression outcome in primary care: A cluster randomized trial. , 2016, Journal of consulting and clinical psychology.

[51]  D P MacKinnon,et al.  Multilevel Mediation Modeling in Group-Based Intervention Studies , 1999, Evaluation review.

[52]  Ahnalee M. Brincks,et al.  A multilevel mediation model of stress and coping for women with HIV and their families. , 2010, Family process.

[53]  D. Cegala,et al.  The impact of patients' participation on physicians' patient-centered communication. , 2009, Patient education and counseling.

[54]  J. Koval,et al.  Influential observations in weighted analyses: examples from the National Longitudinal Survey of Children and Youth (NLSCY). , 2005, Chronic diseases in Canada.

[55]  A. Harvey,et al.  Evidence-based psychological treatments for mental disorders: modifiable barriers to access and possible solutions. , 2015, Behaviour research and therapy.

[56]  T. Chalder,et al.  Establishing how psychological therapies work: the importance of mediation analysis , 2016, Journal of mental health.

[57]  Kosuke Imai,et al.  Experimental designs for identifying causal mechanisms , 2013 .

[58]  Laura M. Stapleton,et al.  Distinguishing Between Cross- and Cluster-Level Mediation Processes in the Cluster Randomized Trial , 2012 .

[59]  Hendrik van den Bussche,et al.  Does GP training in depression care affect patient outcome? - A systematic review and meta-analysis , 2012, BMC Health Services Research.

[60]  David Rindskopf,et al.  Using Multilevel Modeling in the Evaluation of Community-Based Treatment Programs , 2001, Multivariate behavioral research.

[61]  M. Sobel.  Asymptotic Confidence Intervals for Indirect Effects in Structural Equation Models , 1982 .

[62]  Jessaca K. Spybrook,et al.  Progress in the past decade: an examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences , 2016 .

[63]  T. VanderWeele,et al.  Mediation and Spillover Effects in Group-Randomized Trials: A Case Study of the 4Rs Educational Intervention , 2013, Journal of the American Statistical Association.

[64]  Dov Eden,et al.  Field Experiments in Organizations , 2017 .

[65]  Ben Kelcey,et al.  Multilevel design of school effectiveness studies in sub-Saharan Africa , 2016 .

[66]  Laura M. Stapleton,et al.  The Performance of Methods to Test Upper-Level Mediation in the Presence of Nonnormal Data , 2008, Multivariate behavioral research.

[67]  Miha Škerlavaj,et al.  Authentic leadership, creativity, and innovation: A multilevel perspective , 2013 .

[68]  Nathan Jones,et al.  Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes , 2016, Evaluation review.

[69]  H. Bloom,et al.  New Empirical Evidence for the Design of Group Randomized Trials in Education , 2010 .

[70]  F. Pesola,et al.  Supporting recovery in patients with psychosis through care by community-based adult mental health teams (REFOCUS): a multisite, cluster, randomised, controlled trial. , 2015, The lancet. Psychiatry.

[71]  Ben Kelcey,et al.  Strategies for Improving Power in School-Randomized Studies of Professional Development , 2013, Evaluation review.