Avenues for Further Research

Overviews of reviews represent a new publication type and a new form of evidence synthesis. They have rapidly gained popularity. The development of their methodology is, however, still in its infancy. We present a bundle of areas where more work is needed in order to make overviews of reviews more valuable and reliable. Firstly, a clear-cut definition of an overview of reviews is needed. Secondly, methods of presenting the results of overviews of reviews need to be further developed. The needs of groups of users should be kept in mind, in this context, including clinicians, patients, and political decision makers. Maintaining a reasonable balance between necessary complexity and an inevitable loss of information from the reviews is a major challenge. A registration of overviews of reviews is called for. All overviews should be gathered in one freely available registry. When registering new overviews of reviews, a special note should be given to the conflicts of interests of the authors. Reporting guidelines for overviews of reviews should be prepared as soon as possible as this area lacks standardization. Furthermore, more attention should be given to different types of overviews of reviews (e.g., comparison of interventions, comparison of populations).

[1]  L. Kjaergard,et al.  Association between competing interests and authors' conclusions: epidemiological study of randomised clinical trials published in the BMJ , 2002, BMJ : British Medical Journal.

[2]  Johannes B Reitsma,et al.  Variation of a test’s sensitivity and specificity with disease prevalence , 2013, Canadian Medical Association Journal.

[3]  D. Pieper,et al.  Methodological approaches in conducting overviews: current state in HTA agencies , 2014, Research synthesis methods.

[4]  A. Oxman,et al.  Overviews of Reviews , 2008 .

[5]  Elizabeth Wager,et al.  Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study , 2006, BMC medicine.

[6]  J. Flynn,et al.  The optimum time to employ telephotoscreening to detect retinopathy of prematurity. , 2000, Transactions of the American Ophthalmological Society.

[7]  S Greenland,et al.  Quantitative methods in the review of epidemiologic literature. , 1987, Epidemiologic reviews.

[8]  A J Sutton,et al.  Meta‐analysis of individual‐ and aggregate‐level data , 2008, Statistics in medicine.

[9]  Harold I Feldman,et al.  Individual patient‐ versus group‐level data meta‐regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head , 2002, Statistics in medicine.

[10]  Richard D Riley,et al.  Meta‐analysis of a binary outcome using individual participant data and aggregate data , 2010, Research synthesis methods.

[11]  G. Ying,et al.  Predictors for the development of referral-warranted retinopathy of prematurity in the telemedicine approaches to evaluating acute-phase retinopathy of prematurity (e-ROP) study. , 2015, JAMA ophthalmology.

[12]  S. Norris,et al.  Conflict of Interest in Clinical Practice Guideline Development: A Systematic Review , 2011, PloS one.

[13]  J. C. Houwelingen,et al.  Bivariate Random Effects Meta-Analysis of ROC Curves , 2008, Medical decision making : an international journal of the Society for Medical Decision Making.

[14]  P. Thieda,et al.  The effect of study sponsorship on a systematically evaluated body of evidence of head-to-head trials was modest: secondary analysis of a systematic review. , 2010, Journal of clinical epidemiology.

[15]  R A Greenes,et al.  Assessment of diagnostic tests when disease verification is subject to selection bias. , 1983, Biometrics.

[16]  Xiao-Hua Zhou,et al.  Maximum likelihood estimators of sensitivity and specificity corrected for verification bias , 1993 .

[17]  M S Pepe,et al.  Using a combination of reference tests to assess the accuracy of a new diagnostic test. , 1999, Statistics in medicine.

[18]  Alastair J J Wood,et al.  Progress and deficiencies in the registration of clinical trials. , 2009, The New England journal of medicine.

[19]  Jeremy Grimshaw,et al.  AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. , 2009, Journal of clinical epidemiology.

[20]  Kehu Yang,et al.  Quality and transparency of overviews of systematic reviews , 2012, Journal of evidence-based medicine.

[21]  Richard D Riley,et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges , 2016, BMJ.

[22]  Gary S Collins,et al.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement , 2015, BMC Medicine.

[23]  C. Glazener,et al.  Desmopressin for nocturnal enuresis in children. , 2002, The Cochrane database of systematic reviews.

[24]  T. Young,et al.  Association between funding source, methodological quality and research outcomes in randomized controlled trials of synbiotics, probiotics and prebiotics added to infant formula: A Systematic Review , 2013, BMC Medical Research Methodology.

[25]  D. Pieper,et al.  Systematic review finds overlapping reviews were not mentioned in every other overview. , 2014, Journal of clinical epidemiology.

[26]  P. Caldwell,et al.  Simple behavioural interventions for nocturnal enuresis in children. , 2013, The Cochrane database of systematic reviews.

[27]  X H Zhou,et al.  Correcting for verification bias in studies of a diagnostic test's accuracy , 1998, Statistical methods in medical research.

[28]  S. Hui,et al.  Evaluation of diagnostic tests without gold standards , 1998, Statistical methods in medical research.

[29]  K. Facey,et al.  Health technology assessment to optimize health technology utilization: Using implementation initiatives and monitoring processes , 2010, International Journal of Technology Assessment in Health Care.

[30]  M. Clarke,et al.  Methodology in conducting a systematic review of systematic reviews of healthcare interventions , 2011, BMC medical research methodology.

[31]  Haitao Chu,et al.  A simple and robust method for multivariate meta‐analysis of diagnostic test accuracy , 2017, Statistics in medicine.

[32]  M S Pepe,et al.  Comparing disease screening tests when true disease status is ascertained only for screen positives. , 2001, Biostatistics.

[33]  Simon G Thompson,et al.  Can meta-analysis help target interventions at individuals most likely to benefit? , 2005, The Lancet.

[34]  L E Moses,et al.  Estimating Diagnostic Accuracy from Multiple Conflicting Reports , 1993, Medical decision making : an international journal of the Society for Medical Decision Making.

[35]  I. Olkin,et al.  Meta-analysis of observational studies in epidemiology - A proposal for reporting , 2000 .

[36]  Jing Ning,et al.  A hybrid model for combining case–control and cohort studies in systematic reviews of diagnostic tests , 2015, Journal of the Royal Statistical Society. Series C, Applied statistics.

[37]  Thomas A Louis,et al.  Random Effects Models in a Meta-Analysis of the Accuracy of Two Diagnostic Tests Without a Gold Standard , 2009, Journal of the American Statistical Association.

[38]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[39]  John P A Ioannidis,et al.  Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. , 2012, Journal of clinical epidemiology.

[40]  Nandini Dendukuri,et al.  Bayesian Meta‐Analysis of the Accuracy of a Test for Tuberculous Pleuritis in the Absence of a Gold Standard Reference , 2012, Biometrics.

[41]  R. Hardy,et al.  The multicenter study of Early Treatment for Retinopathy of Prematurity (ETROP). , 2001, Ophthalmology.

[42]  Margaret Sullivan Pepe,et al.  Insights into latent class analysis of diagnostic test performance. , 2007, Biostatistics.

[43]  P. Vineis,et al.  Conflicts of interest matter and awareness is needed , 2015, Journal of Epidemiology & Community Health.

[44]  Loes C M Bertens,et al.  Value of composite reference standards in diagnostic research , 2013, BMJ.

[45]  Peter C Gøtzsche,et al.  Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review , 2006, BMJ : British Medical Journal.

[46]  D. Moher,et al.  Guidance for Developers of Health Research Reporting Guidelines , 2010, PLoS medicine.

[47]  M. Schulze,et al.  Financial Conflicts of Interest and Reporting Bias Regarding the Association between Sugar-Sweetened Beverages and Weight Gain: A Systematic Review of Systematic Reviews , 2013, PLoS medicine.

[48]  S. Norris,et al.  Conflicts of Interest among Authors of Clinical Practice Guidelines for Glycemic Control in Type 2 Diabetes Mellitus , 2013, PloS one.

[49]  C. Adams,et al.  Systematic Overview of Cochrane Reviews for Anticholinergic Effects of Antipsychotic Drugs , 2009, Journal of clinical psychopharmacology.

[50]  D B Schaffer,et al.  Prognostic factors in the natural course of retinopathy of prematurity. The Cryotherapy for Retinopathy of Prematurity Cooperative Group. , 1993, Ophthalmology.

[51]  Jing Ning,et al.  A composite likelihood method for bivariate meta-analysis in diagnostic systematic reviews , 2017, Statistical methods in medical research.

[52]  J. Ioannidis,et al.  Head-to-head randomized trials are mostly industry sponsored and almost always favor the industry sponsor. , 2015, Journal of clinical epidemiology.

[53]  E. Ernst,et al.  Adverse effects of herbal medicines: an overview of systematic reviews. , 2013, Clinical medicine.

[54]  J. Schneider,et al.  Endometrial carcinoma: Assessment of myometrial invasion with plain and gadolinium‐enhanced MR imaging , 2000, Journal of magnetic resonance imaging : JMRI.

[55]  K J M Janssen,et al.  Multiple imputation to correct for partial verification bias revisited , 2008, Statistics in medicine.

[56]  D. Moher,et al.  The nuts and bolts of PROSPERO: an international prospective register of systematic reviews , 2012, Systematic Reviews.

[57]  Karina D. Torralba,et al.  Association of industry funding with the outcome and quality of randomized controlled trials of drug therapy for rheumatoid arthritis. , 2012, Arthritis and rheumatism.

[58]  S. Walter,et al.  Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data , 2002, Statistics in medicine.

[59]  Richard D Riley,et al.  Meta‐analysis of diagnostic test studies using individual patient data and aggregate data , 2008, Statistics in medicine.

[60]  Haitao Chu,et al.  Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. , 2006, Journal of clinical epidemiology.

[61]  Roger M Harbord,et al.  A unification of models for meta-analysis of diagnostic accuracy studies. , 2007, Biostatistics.

[62]  H C Van Houwelingen,et al.  A bivariate approach to meta-analysis. , 1993, Statistics in medicine.

[63]  Marilyn J. Field,et al.  Conflict of Interest in Medical Research, Education, and Practice , 2009 .

[64]  F. Chiappelli,et al.  From Systematic Reviews to Clinical Recommendations for Evidence-Based Health Care: Validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for Grading of Clinical Relevance , 2010, The open dentistry journal.

[65]  P. Gustafson On Model Expansion, Model Contraction, Identifiability and Prior Information: Two Illustrative Scenarios Involving Mismeasured Variables , 2005 .

[66]  G. Guyatt,et al.  Adverse effects of biologics: a network meta-analysis and Cochrane overview. , 2011, The Cochrane database of systematic reviews.

[67]  P. Albert,et al.  A Cautionary Note on the Robustness of Latent Class Models for Estimating Diagnostic Error without a Gold Standard , 2004, Biometrics.

[68]  Haitao Chu,et al.  A hybrid Bayesian hierarchical model combining cohort and case–control studies for meta-analysis of diagnostic tests: Accounting for partial verification bias , 2016, Statistical methods in medical research.

[69]  Christian Gluud,et al.  Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? , 2003, JAMA.

[70]  S. Thompson,et al.  How should meta‐regression analyses be undertaken and interpreted? , 2002, Statistics in medicine.

[71]  Dawid Pieper,et al.  Up-to-dateness of reviews is often neglected in overviews: a systematic review. , 2014, Journal of clinical epidemiology.

[72]  C. Gilbert Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. , 2008, Early human development.

[73]  Sara Schroter,et al.  Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. , 2006, JAMA.

[74]  Alenka Tančič Grum,et al.  Best Practice Elements of Multilevel Suicide Prevention Strategies , 2011, Crisis.

[75]  Peter C Gøtzsche,et al.  Industry-supported meta-analyses compared with meta-analyses with non-profit or no support : Differences in methodological quality and conclusions , 2015 .

[76]  A R Feinstein,et al.  Misguided efforts and future challenges for research on “diagnostic tests” , 2002, Journal of epidemiology and community health.

[77]  L. Joseph,et al.  Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. , 1995, American journal of epidemiology.

[78]  L E Moses,et al.  Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. , 1993, Statistics in medicine.

[79]  T. Iwasaka,et al.  MR imaging in endometrial carcinoma as a diagnostic tool for the absence of myometrial invasion. , 2006, Gynecologic oncology.

[80]  Haitao Chu,et al.  A unification of models for meta‐analysis of diagnostic accuracy studies without a gold standard , 2015, Biometrics.

[81]  Dawid Pieper,et al.  Overviews of reviews often have limited rigor: a systematic review. , 2012, Journal of clinical epidemiology.

[82]  Johannes B. Reitsma,et al.  Individual Participant Data (IPD) Meta-analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use , 2015, PLoS medicine.

[83]  Chuong B. Do,et al.  Serum Iron Levels and the Risk of Parkinson Disease: A Mendelian Randomization Study , 2013, PLoS medicine.

[84]  Denise Thomson,et al.  A Descriptive Analysis of Overviews of Reviews Published between 2000 and 2011 , 2012, PloS one.

[85]  D. Cook,et al.  A guide to interpreting discordant systematic reviews. , 1997, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[86]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement , 2009, BMJ : British Medical Journal.

[87]  Juan Lu,et al.  Predicting Outcome after Traumatic Brain Injury: Development and International Validation of Prognostic Scores Based on Admission Characteristics , 2008, PLoS medicine.

[88]  Paul Landais,et al.  Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors. , 2004, Journal of clinical epidemiology.

[89]  Ofer Harel,et al.  Multiple imputation for correcting verification bias , 2006, Statistics in medicine.

[90]  D. Cherkin,et al.  Complementary and alternative therapies for back pain II. , 2010, Evidence report/technology assessment.

[91]  A. Feinstein,et al.  Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. , 1978, The New England journal of medicine.

[92]  D. Moher,et al.  Declaration of transparency for each research article , 2013, BMJ.

[93]  Paula R Williamson,et al.  Investigating heterogeneity in an individual patient data meta‐analysis of time to event outcomes , 2005, Statistics in medicine.

[94]  Bart W. Koes,et al.  A systematic review on the effectiveness of physical and rehabilitation interventions for chronic non-specific low back pain , 2010, European Spine Journal.

[95]  Lowell A Goldsmith,et al.  Picking your peers. , 2006, The Journal of investigative dermatology.

[96]  R. Lilford,et al.  Efficacy of Adjuvant Chemotherapy after Surgery when Considered over all Cancer Types: A Synthesis of Meta-Analyses , 2012, Annals of Surgical Oncology.

[97]  D. Cheuk,et al.  Complementary and miscellaneous interventions for nocturnal enuresis in children. , 2011, The Cochrane database of systematic reviews.

[98]  Johannes B Reitsma,et al.  Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. , 2005, Journal of clinical epidemiology.

[99]  Justin Starren,et al.  Telemedicine for retinopathy of prematurity diagnosis: evaluation and challenges. , 2009, Survey of ophthalmology.

[100]  R. Riley,et al.  Meta-analysis of individual participant data: rationale, conduct, and reporting , 2010, BMJ : British Medical Journal.

[101]  E W Steyerberg,et al.  See Blockindiscussions, Blockinstats, Blockinand Blockinauthor Blockinprofiles Blockinfor Blockinthis Blockinpublication Prognostic Blockinmodels Blockinbased Blockinon Blockinliterature Blockinand Individual Blockinpatient Blockindata Blockinin Blockinlogistic Blockinregression Analysis Article Blo , 2022 .

[102]  M. Tan,et al.  Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. , 1996, Biometrics.

[103]  J. Starr,et al.  The effect of funding sources on donepezil randomised controlled trial outcome: a meta-analysis , 2014, BMJ Open.

[104]  R. Riera,et al.  Overview of systematic reviews - a new type of study. Part II. , 2014, Sao Paulo medical journal = Revista paulista de medicina.

[105]  Enrico Coiera,et al.  Financial Conflicts of Interest and Conclusions About Neuraminidase Inhibitors for Influenza , 2014, Annals of Internal Medicine.

[106]  Emil H Schemitsch,et al.  Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. , 2004, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[107]  Karla Hemming,et al.  Scientific hypotheses can be tested by comparing the effects of one treatment over many diseases in a systematic review. , 2014, Journal of clinical epidemiology.

[108]  B. Prediger,et al.  Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties. , 2015, Journal of clinical epidemiology.

[109]  Karel G M Moons,et al.  A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta‐analysis , 2013, Statistics in medicine.

[110]  Johannes B Reitsma,et al.  Evidence of bias and variation in diagnostic accuracy studies , 2006, Canadian Medical Association Journal.

[111]  Dawid Pieper,et al.  State of evidence on the relationship between high-volume hospitals and outcomes in surgery: a systematic review of systematic reviews. , 2013, Journal of the American College of Surgeons.

[112]  S. Walter,et al.  Estimating the error rates of diagnostic tests. , 1980, Biometrics.

[113]  Veronica Yank,et al.  Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study , 2007, BMJ : British Medical Journal.

[114]  A. Deshpande,et al.  Drugs for nocturnal enuresis in children (other than desmopressin and tricyclics). , 2012, The Cochrane database of systematic reviews.

[115]  C. Glazener,et al.  Alarm interventions for nocturnal enuresis in children. , 2005, The Cochrane database of systematic reviews.

[116]  Richard D Riley,et al.  Developing and validating risk prediction models in an individual participant data meta-analysis , 2014, BMC Medical Research Methodology.

[117]  Theo Stijnen,et al.  Advanced methods in meta‐analysis: multivariate approach and meta‐regression , 2002, Statistics in medicine.

[118]  Yvonne Vergouwe,et al.  Incorporating published univariable associations in diagnostic and prognostic modeling , 2012, BMC Medical Research Methodology.

[119]  Harry Joe,et al.  Composite Likelihood Methods , 2012 .

[120]  I Olkin,et al.  Comparison of effect estimates from a meta-analysis of summary data from published studies and from a meta-analysis using individual patient data for ovarian cancer studies. , 1997, American journal of epidemiology.

[121]  P C Lambert,et al.  A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis. , 2002, Journal of clinical epidemiology.

[122]  R. Reznek,et al.  Evaluation of endometrial carcinoma on magnetic resonance imaging , 2006, International Journal of Gynecologic Cancer.

[123]  Theo Stijnen,et al.  The binomial distribution of meta-analysis was preferred to model within-study variability. , 2008, Journal of clinical epidemiology.

[124]  Haitao Chu,et al.  Meta‐analysis of diagnostic accuracy studies accounting for disease prevalence: Alternative parameterizations and model selection , 2009, Statistics in medicine.

[125]  Deborah R Erlich,et al.  Conflicts of interest and the quality of recommendations in clinical guidelines. , 2013, Journal of evaluation in clinical practice.

[126]  Nancy R. Cook,et al.  Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[127]  A D Oxman,et al.  Systematic reviews of the effectiveness of quality improvement strategies and programmes , 2003, Quality & safety in health care.

[128]  Lisa Bero,et al.  Factors Associated with Findings of Published Trials of Drug–Drug Comparisons: Why Some Statins Appear More Efficacious than Others , 2007, PLoS medicine.

[129]  L. Joseph,et al.  Bayesian Approaches to Modeling the Conditional Dependence Between Multiple Diagnostic Tests , 2001, Biometrics.

[130]  H. Schünemann,et al.  Author's specialty and conflicts of interest contribute to conflicting guidelines for screening mammography. , 2012, Journal of clinical epidemiology.

[131]  Karel G M Moons,et al.  Aggregating published prediction models with individual participant data: a comparison of different approaches , 2012, Statistics in medicine.

[132]  S. Oliver,et al.  Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ , 2012, BMC Medical Research Methodology.