Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review

BackgroundThe COMET (Core Outcome Measures in Effectiveness Trials) Initiative is developing a publicly accessible online resource to collate the knowledge base for core outcome set development (COS) and the applied work from different health conditions. Ensuring that the database is as comprehensive as possible and keeping it up to date are key to its value for users. This requires the development and application of an optimal, multi-faceted search strategy to identify relevant material. This paper describes the challenges of designing and implementing such a search, outlining the development of the search strategy for studies of COS development, and, in turn, the process for establishing a database of COS.MethodsWe investigated the performance characteristics of this strategy including sensitivity, precision and numbers needed to read. We compared the contribution of databases towards identifying included studies to identify the best combination of methods to retrieve all included studies.ResultsRecall of the search strategies ranged from 4% to 87%, and precision from 0.77% to 1.13%. MEDLINE performed best in terms of recall, retrieving 216 (87%) of the 250 included records, followed by Scopus (44%). The Cochrane Methodology Register found just 4% of the included records. MEDLINE was also the database with the highest precision. The number needed to read varied between 89 (MEDLINE) and 130 (SCOPUS).ConclusionsWe found that two databases and hand searching were required to locate all of the studies in this review. MEDLINE alone retrieved 87% of the included studies, but actually 97% of the included studies were indexed on MEDLINE. The Cochrane Methodology Register did not contribute any records that were not found in the other databases, and will not be included in our future searches to identify studies developing COS. SCOPUS had the lowest precision rate (0.77) and highest number needed to read (130). In future COMET searches for COS a balance needs to be struck between the work involved in screening large numbers of records, the frequency of the searching and the likelihood that eligible studies will be identified by means other than the database searches.

[1]  M. Clarke,et al.  Handsearching versus electronic searching to identify reports of randomized trials. , 2002, The Cochrane database of systematic reviews.

[2]  S. Brophy,et al.  Interventions for latent autoimmune diabetes (LADA) in adults. , 2011, The Cochrane database of systematic reviews.

[3]  Katherine Payne,et al.  Patient Empowerment in Clinical Genetics Services , 2008, Journal of health psychology.

[4]  David Tovey,et al.  The impact of Cochrane Reviews. , 2010, The Cochrane database of systematic reviews.

[5]  Katherine Payne,et al.  Outcome measurement in clinical genetics services: a systematic review of validated measures. , 2008, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[6]  Michele Tarsilla.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, Journal of MultiDisciplinary Evaluation.

[7]  Katherine Payne,et al.  Improving Service Evaluation in Clinical Genetics: Identifying Effects of Genetic Diseases on Individuals and Families , 2007, Journal of Genetic Counseling.

[8]  M Clarke.  Searching MEDLINE for randomised trials. , 1993, BMJ.

[9]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration , 2013 .

[10]  P. Williamson,et al.  Bias in meta‐analysis due to outcome variable selection within studies , 2000 .

[11]  P. Richardson,et al.  Identifying randomized controlled trials of cognitive therapy for depression: comparing the efficiency of Embase, Medline and PsycINFO bibliographic databases. , 1999, The British journal of medical psychology.

[12]  Barbarie Hill.  Comparison of journal title coverage between CINAHL and Scopus. , 2009, Journal of the Medical Library Association : JMLA.

[13]  R. Haynes,et al.  Comparison of top-performing search strategies for detecting clinically sound treatment studies and systematic reviews in MEDLINE and EMBASE. , 2006, Journal of the Medical Library Association : JMLA.

[14]  G Dunn,et al.  The Genetic Counseling Outcome Scale: a new patient‐reported outcome measure for clinical genetics services , 2011, Clinical genetics.

[15]  Anne Eisinga,et al.  Enhancing access to reports of randomized trials published world-wide – the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library , 2008, Emerging themes in epidemiology.

[16]  Steve McDonald,et al.  Development of the Cochrane Collaboration’s Central Register of Controlled Clinical Trials , 2002, Evaluation & the health professions.

[17]  A. Miller,et al.  Reporting results of cancer treatment , 1981, Cancer.

[18]  R. Macleod,et al.  The emotional effects of genetic diseases: Implications for clinical genetics , 2007, American journal of medical genetics. Part A.

[19]  Graham Dunn,et al.  Empowerment: qualitative underpinning of a new clinical genetics-specific patient-reported outcome , 2011, European Journal of Human Genetics.

[20]  P. Williamson,et al.  Using the Delphi Technique to Determine Which Outcomes to Measure in Clinical Trials: Recommendations for the Future Based on a Systematic Review of Existing Studies , 2011, PLoS medicine.

[21]  Elizabeth Gargon,et al.  Can a core outcome set improve the quality of systematic reviews? – a survey of the Co-ordinating Editors of Cochrane review groups , 2013, Trials.

[22]  W. Velicer,et al.  Biochemical verification of tobacco use and cessation. , 2002, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[23]  Thomas E Schlaepfer,et al.  Deep brain stimulation and the neuroethics of responsible publishing: when one is not enough. , 2010, JAMA.

[24]  Paula R Williamson,et al.  A Systematic Review of Studies That Aim to Determine Which Outcomes to Measure in Clinical Trials in Children , 2008, PLoS medicine.

[25]  Jane M Blazeby,et al.  Developing core outcome sets for clinical trials: issues to consider , 2012, Trials.

[26]  Paula R. Williamson,et al.  Choosing Important Health Outcomes for Comparative Effectiveness Research: A Systematic Review , 2014, PloS one.

[27]  Douglas G Altman,et al.  The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews , 2010, BMJ : British Medical Journal.

[28]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.