Addressing the estimation of standard errors in fixed effects meta‐analysis

Standard methods for fixed effects meta‐analysis assume that standard errors for study‐specific estimates are known, not estimated. While the impact of this simplifying assumption has been shown in a few special cases, its general impact is not well understood, nor are general‐purpose tools available for inference under more realistic assumptions. In this paper, we aim to elucidate the impact of using estimated standard errors in fixed effects meta‐analysis, showing why it does not go away in large samples and quantifying how badly miscalibrated standard inference will be if it is ignored. We also show the important role of a particular measure of heterogeneity in this miscalibration. These developments lead to confidence intervals for fixed effects meta‐analysis with improved performance for both location and scale parameters.

[1]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[2]  H. D. Patterson,et al.  Recovery of inter-block information when block sizes are unequal , 1971 .

[3]  R. Fisher.  014: On the "Probable Error" of a Coefficient of Correlation Deduced from a Small Sample. , 1921 .

[4]  E. Mammen.  When does bootstrap work , 1992 .

[5]  Ingram Olkin,et al.  Why add anything to nothing? The arcsine difference as a measure of treatment effect in meta‐analysis with zero cells , 2009, Statistics in medicine.

[6]  Li Shi,et al.  The bias of the commonly-used estimate of variance in meta-analysis , 1994 .

[7]  Stephen Senn,et al.  (www.interscience.wiley.com) DOI: 10.1002/sim.2639 Trying to be precise about vagueness , 2022 .

[8]  Larry V. Hedges,et al.  Analyzing effect sizes: Fixed-effects models , 2009 .

[9]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[10]  Joachim Hartung,et al.  Some Corrections of the Significance Level in Meta - Analysis , 1998 .

[11]  Prem K. Goel,et al.  Information Measures and Bayesian Hierarchical Models , 1983 .

[12]  Larry V. Hedges,et al.  A random effects model for effect sizes , 1983 .

[13]  D. Zeng,et al.  On the relative efficiency of using summary statistics versus individual-level data in meta-analysis. , 2010, Biometrika.

[14]  A. Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[15]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[16]  David R. Jones,et al.  How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS , 2005, Statistics in medicine.

[17]  Andrew Gelman,et al.  Discussion: Difficulties in making inferences about scientific truth from distributions of published p-values. , 2014, Biostatistics.

[18]  I Marshall,et al.  Zinc for the common cold. , 1999, The Cochrane database of systematic reviews.

[19]  John T. Kent,et al.  Confidence intervals for the noncentral chi-squared distribution , 1995 .

[20]  D. Harville.  Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems , 1977 .

[21]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[22]  P. Gustafson,et al.  Conservative prior distributions for variance parameters in hierarchical models , 2006 .

[23]  M. Borenstein.  Effect sizes for continuous data. , 2009 .

[24]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[25]  S G Thompson,et al.  A likelihood approach to meta-analysis with random effects. , 1996, Statistics in medicine.

[26]  D. R. Cox,et al.  Interpretation of interaction: A review , 2007, 0712.1106.

[27]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[28]  Julian P T Higgins,et al.  A re‐evaluation of fixed effect(s) meta‐analysis , 2018 .

[29]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[30]  M. A. Best.  Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[31]  S. Thompson,et al.  Detecting and describing heterogeneity in meta-analysis. , 1998, Statistics in medicine.

[32]  B. Sinha,et al.  Statistical Meta-Analysis with Applications , 2008 .

[33]  Stephen W. Raudenbush,et al.  Analyzing effect sizes: Random-effects models. , 2009 .

[34]  David J Spiegelhalter,et al.  A re-evaluation of random-effects meta-analysis , 2009, Journal of the Royal Statistical Society. Series A,.

[35]  D. Sackett,et al.  Cochrane Collaboration , 1994, BMJ.

[36]  Wolfgang Viechtbauer,et al.  Confidence intervals for the amount of heterogeneity in meta‐analysis , 2007, Statistics in medicine.

[37]  J. Higgins.  Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[38]  Kurex Sidik,et al.  Simple heterogeneity variance estimation for meta‐analysis , 2005 .

[39]  L. Hedges,et al.  The power of statistical tests in meta-analysis. , 2001, Psychological methods.

[40]  J. Hartung,et al.  On tests of the overall treatment effect in meta‐analysis with normally distributed responses , 2001, Statistics in medicine.

[41]  P. Patnaik.  THE NON-CENTRAL χ2- AND F-DISTRIBUTIONS AND THEIR APPLICATIONS , 1949 .

[42]  Wolfgang Viechtbauer,et al.  Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model , 2005 .

[43]  J. Cochran.  What is the bootstrap? , 2019, Significance.

[44]  R Peto,et al.  Why do we need systematic overviews of randomized trials? , 1987, Statistics in medicine.

[45]  S. Pocock,et al.  Can meta-analyses be trusted? , 1991, The Lancet.

[46]  F Mosteller,et al.  Some Statistical Methods for Combining Experimental Results , 1990, International Journal of Technology Assessment in Health Care.

[47]  Hannah R Rothstein,et al.  A basic introduction to fixed‐effect and random‐effects models for meta‐analysis , 2010, Research synthesis methods.

[48]  Jack Kaplan,et al.  Estimators for the One-Way Random Effects Model with Unequal Error Variances , 1981 .