Methods for evidence synthesis in the case of very few studies

In systematic reviews, meta‐analyses are routinely applied to summarize the results of the relevant studies for a specific research question. If one can assume that in all studies the same true effect is estimated, the application of a meta‐analysis with common effect (commonly referred to as fixed‐effect meta‐analysis) is adequate. If between‐study heterogeneity is expected to be present, the method of choice is a meta‐analysis with random effects. The widely used DerSimonian and Laird method for meta‐analyses with random effects has been criticized due to its unfavorable statistical properties, especially in the case of very few studies. A working group of the Cochrane Collaboration recommended the use of the Knapp‐Hartung method for meta‐analyses with random effects. However, as heterogeneity cannot be reliably estimated if only very few studies are available, the Knapp‐Hartung method, while correctly accounting for the corresponding uncertainty, has very low power. Our aim is to summarize possible methods to perform meaningful evidence syntheses in the situation with only very few (ie, 2‐4) studies. Some general recommendations are provided on which method should be used when. Our recommendations are based on the existing literature on methods for meta‐analysis with very few studies and consensus of the authors. The recommendations are illustrated by 2 examples coming from dossier assessments of the Institute for Quality and Efficiency in Health Care.

[1]  Chukiat Viwatwongkasem,et al.  Some general points in estimating heterogeneity variance with the DerSimonian-Laird estimator. , 2002, Biostatistics.

[2]  J D Emerson,et al.  Combining estimates of the odds ratio: the state of the art , 1994, Statistical methods in medical research.

[3]  Dankmar Böhning,et al.  Meta-analysis of Binary Data Using Profile Likelihood , 2008 .

[4]  George F Borm,et al.  The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method , 2014, BMC Medical Research Methodology.

[5]  Kalliopi Mylona,et al.  Meta‐analysis of clinical trials with rare events , 2015, Biometrical journal. Biometrische Zeitschrift.

[6]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, International Coaching Psychology Review.

[7]  F Mosteller,et al.  Some Statistical Methods for Combining Experimental Results , 1990, International Journal of Technology Assessment in Health Care.

[8]  A. Großhennig,et al.  No solution yet for combining two independent studies in the presence of heterogeneity , 2015, Statistics in medicine.

[9]  D J Spiegelhalter,et al.  Bayesian approaches to random-effects meta-analysis: a comparative study. , 1995, Statistics in medicine.

[10]  Julian P T Higgins,et al.  An empirical comparison of heterogeneity variance estimators in 12 894 meta‐analyses , 2015, Research synthesis methods.

[11]  Mark C Simmonds,et al.  A general framework for the use of logistic regression models in meta-analysis , 2016, Statistical methods in medical research.

[12]  N Breslow,et al.  Generalized linear mixed models for meta-analysis. , 1999, Statistics in medicine.

[13]  John Mandel,et al.  Consensus Values and Weighting Factors. , 1982, Journal of research of the National Bureau of Standards.

[14]  C. Di Pietrantonj.  Four‐fold table cell frequencies imputation in meta analysis , 2006, Statistics in Medicine.

[15]  Alan Agresti,et al.  Categorical Data Analysis , 2003 .

[16]  Jonathan J Deeks,et al.  Much ado about nothing: a comparison of the performance of meta‐analytical methods with rare events , 2007, Statistics in medicine.

[17]  Minge Xie,et al.  Exact Meta-Analysis Approach for Discrete Data and its Application to 2 × 2 Tables With Rare Events , 2014, Journal of the American Statistical Association.

[18]  Kurex Sidik,et al.  On Constructing Confidence Intervals for a Standardized Mean Difference in Meta-analysis , 2003 .

[19]  Guido Knapp,et al.  Improved tests for a random effects meta‐regression with a single covariate , 2003, Statistics in medicine.

[20]  Ralf Bender,et al.  Methods to estimate the between‐study variance and its uncertainty in meta‐analysis† , 2015, Research synthesis methods.

[21]  Cynthia D Mulrow,et al.  Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change , 2014, Annals of Internal Medicine.

[22]  Theo Stijnen,et al.  Random effects meta‐analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data , 2010, Statistics in medicine.

[23]  Joachim Hartung,et al.  An Alternative Method for Meta‐Analysis , 1999 .

[24]  Bayesian Estimation and Testing in Random Effects Meta-Analysis of Rare Binary Adverse Events , 2016, Statistics in biopharmaceutical research.

[25]  Rebecca M. Turner,et al.  Predictive distributions were developed for the extent of heterogeneity in meta-analyses of continuous outcome data , 2015, Journal of clinical epidemiology.

[26]  T. Raghunathan,et al.  Analysis of binary data from a multicentre clinical trial , 1993 .

[27]  O Kuss,et al.  Statistical methods for meta‐analyses including information from studies without any events—add nothing to nothing and succeed nevertheless , 2015, Statistics in medicine.

[28]  M. A. Best.  Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[29]  David J Spiegelhalter,et al.  A re-evaluation of random-effects meta-analysis , 2009, Journal of the Royal Statistical Society. Series A,.

[30]  Gerta Rücker,et al.  Hartung–Knapp method is not always conservative compared with fixed‐effect meta‐analysis , 2016, Statistics in medicine.

[31]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[32]  Simon G Thompson,et al.  Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews , 2012, International journal of epidemiology.

[33]  Julian P T Higgins,et al.  A re‐evaluation of fixed effect(s) meta‐analysis , 2018 .

[34]  T. Friede,et al.  Discrete Approximation of a Mixture Distribution via Restricted Divergence , 2016, 1602.04060.

[35]  N Breslow,et al.  Approximate hierarchical modelling of discrete data in epidemiology , 1998, Statistical methods in medical research.

[36]  Tim Friede,et al.  Meta‐analysis of two studies in the presence of heterogeneity with applications in rare diseases , 2016, Biometrical journal. Biometrische Zeitschrift.

[37]  Sujuan Gao.  Combining binomial data using the logistic normal model , 2004 .

[38]  T. Friede,et al.  Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies , 2015, BMC Medical Research Methodology.

[39]  Dan Jackson,et al.  Predictive distributions for between-study heterogeneity and simple methods for their application in Bayesian meta-analysis , 2014, Statistics in medicine.

[40]  Alexander J Sutton,et al.  What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. , 2004, Statistics in medicine.

[41]  Tim Friede,et al.  Meta‐analysis of few small studies in orphan diseases , 2016, Research synthesis methods.

[42]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[43]  R. Riley,et al.  Meta-analysis of individual participant data: rationale, conduct, and reporting , 2010, BMJ : British Medical Journal.

[44]  Annamaria Guolo,et al.  Random-effects meta-analysis: the number of studies matters , 2017, Statistical methods in medical research.