Estimation of the predictive power of the model in mixed-effects meta-regression: A simulation study.

Several methods are available to estimate the total and residual amount of heterogeneity in meta-analysis, leading to different alternatives when estimating the predictive power in mixed-effects meta-regression models using the formula proposed by Raudenbush (1994, 2009). In this paper, a simulation study was conducted to compare the performance of seven estimators of these parameters under various realistic scenarios in psychology and related fields. Our results suggest that the number of studies (k) exerts the most important influence on the accuracy of the results, and that precise estimates of the heterogeneity variances and the model predictive power can only be expected with at least 20 and 40 studies, respectively. Increases in the average within-study sample size (N¯) also improved the results for all estimators. Some differences among the accuracy of the estimators were observed, especially under adverse (small k and N¯) conditions, while the results for the different methods tended to convergence for more optimal scenarios.

[1]  S. Thompson,et al.  How should meta‐regression analyses be undertaken and interpreted? , 2002, Statistics in medicine.

[2]  Wolfgang Viechtbauer,et al.  Accounting for Heterogeneity via Random-Effects Models and Moderator Analyses in Meta-Analysis , 2007 .

[3]  In-Sue Oh,et al.  Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. , 2009, The British journal of mathematical and statistical psychology.

[4]  Harris Cooper,et al.  The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. , 2009, Psychological methods.

[5]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[6]  Janet Shibley Hyde,et al.  Cross-national patterns of gender differences in mathematics: a meta-analysis. , 2010, Psychological bulletin.

[7]  Larry V. Hedges,et al.  Analyzing effect sizes: Fixed-effects models , 2009 .

[8]  Kurex Sidik,et al.  A Note on Variance Estimation in Random Effects Meta-Regression , 2005, Journal of biopharmaceutical statistics.

[9]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[10]  Stephen W. Raudenbush,et al.  Random effects models. , 1994 .

[11]  Julio Sánchez-Meca,et al.  Confidence intervals for the overall effect size in random-effects meta-analysis. , 2008, Psychological methods.

[12]  Heinz Holling,et al.  Nonparametric estimation of heterogeneity variance for the standardised difference used in meta-analysis , 2000 .

[13]  Wolfgang Viechtbauer,et al.  Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model , 2005 .

[14]  Michael C. Rodriguez,et al.  Meta-analysis of coefficient alpha. , 2006, Psychological methods.

[15]  Therese D. Pigott,et al.  An alternative to R2 for assessing linear models of effect size , 2010, Research synthesis methods.

[16]  J. Sánchez-Meca,et al.  TESTING CONTINUOUS MODERATORS IN META-ANALYSIS : A COMPARISON OF PROCEDURES , 1998 .

[17]  Julio Sánchez-Meca,et al.  Weighting by Inverse Variance or by Sample Size in Random-Effects Meta-Analysis , 2010 .

[18]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[19]  Andreas Rosenblad,et al.  Introduction to Meta‐Analysis by Michael Borenstein, Larry V. Hedges, Julian P.T. Higgins, Hannah R. Rothstein , 2009 .

[20]  Kurex Sidik,et al.  A comparison of heterogeneity variance estimators in combining results of studies , 2007, Statistics in medicine.

[21]  Wolfgang Viechtbauer,et al.  Hypothesis tests for population heterogeneity in meta-analysis. , 2007, The British journal of mathematical and statistical psychology.

[22]  Stephen W. Raudenbush,et al.  Analyzing effect sizes: Random-effects models. , 2009 .

[23]  S G Thompson,et al.  Systematic Review: Why sources of heterogeneity in meta-analysis should be investigated , 1994, BMJ.

[24]  Diane M. Tomasic,et al.  Meta-Analysis: A Comparison of Approaches , 2005 .

[25]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[26]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[27]  Guido Knapp,et al.  Assessing the Amount of Heterogeneity in Random‐Effects Meta‐Analysis , 2006, Biometrical journal. Biometrische Zeitschrift.

[28]  J. Sánchez-Meca,et al.  A comparison of procedures to test for moderators in mixed-effects meta-regression models. , 2015, Psychological methods.

[29]  Effects Meta-Analysis,et al.  Weighting by Inverse Variance or by Sample Size in Random , 2010 .

[30]  D. P. Gaver,et al.  Combining Information: Statistical Issues and Opportunities for Research , 1993 .

[31]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[32]  Patrick Onghena,et al.  Multilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures , 2003 .

[33]  P Gangadharan,et al.  Statistical considerations. , 1973, Indian journal of cancer.

[34]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[35]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[36]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[37]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[38]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[39]  Simon G. Thompson,et al.  Meta‐Analysis of Clinical Trials , 2005 .

[40]  Larry V Hedges,et al.  The power of statistical tests for moderators in meta-analysis. , 2004, Psychological methods.

[41]  Robert I. Jennrich,et al.  Newton-Raphson and Related Algorithms for Maximum Likelihood Variance Component Estimation , 1976 .

[42]  S. Sharp,et al.  Explaining heterogeneity in meta-analysis: a comparison of methods. , 1999 .