Sensitivity and Predictive Value of 15 PubMed Search Strategies to Answer Clinical Questions Rated Against Full Systematic Reviews

Background Clinicians perform searches in PubMed daily, but retrieving relevant studies is challenging due to the rapid expansion of medical knowledge. Little is known about the performance of search strategies when they are applied to answer specific clinical questions. Objective To compare the performance of 15 PubMed search strategies in retrieving relevant clinical trials on therapeutic interventions. Methods We used Cochrane systematic reviews to identify relevant trials for 30 clinical questions. Search terms were extracted from the abstract using a predefined procedure based on the population, interventions, comparison, outcomes (PICO) framework and combined into queries. We tested 15 search strategies that varied in their query (PIC or PICO), use of PubMed’s Clinical Queries therapeutic filters (broad or narrow), search limits, and PubMed links to related articles. We assessed sensitivity (recall) and positive predictive value (precision) of each strategy on the first 2 PubMed pages (40 articles) and on the complete search output. Results The performance of the search strategies varied widely according to the clinical question. Unfiltered searches and those using the broad filter of Clinical Queries produced large outputs and retrieved few relevant articles within the first 2 pages, resulting in a median sensitivity of only 10%–25%. In contrast, all searches using the narrow filter performed significantly better, with a median sensitivity of about 50% (all P < .001 compared with unfiltered queries) and positive predictive values of 20%–30% (P < .001 compared with unfiltered queries). This benefit was consistent for most clinical questions. Searches based on related articles retrieved about a third of the relevant studies. Conclusions The Clinical Queries narrow filter, along with well-formulated queries based on the PICO framework, provided the greatest aid in retrieving relevant clinical trials within the 2 first PubMed pages. These results can help clinicians apply effective strategies to answer their questions at the point of care.

[1]  Arjen Hoogendam,et al.  Evaluation of PubMed filters used for evidence-based searching: validation using relative recall. , 2009, Journal of the Medical Library Association : JMLA.

[2]  Amit X. Garg,et al.  Searching for medical information online: a survey of Canadian nephrologists. , 2011, Journal of nephrology.

[3]  Cindy Farquhar,et al.  3 The Cochrane Library , 1996 .

[4]  A R Jadad,et al.  Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. , 1998, JAMA.

[5]  Alessandro Liberati,et al.  Speed of updating online evidence based point of care summaries: prospective cohort analysis , 2011, BMJ : British Medical Journal.

[6]  L. Gruppen,et al.  A Controlled Comparison Study of the Efficacy of Training Medical Students in Evidence-Based Medicine Literature Searching Skills , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[7]  B. Haynes.  Of studies, syntheses, synopses, summaries, and systems: the "5S" evolution of information services for evidence-based healthcare decisions. , 2006, Evidence-based nursing.

[8]  Martin Dawes,et al.  Information Needs and Information-Seeking Behavior of Primary Care Physicians , 2007, The Annals of Family Medicine.

[9]  R. Thiele,et al.  Speed, accuracy, and confidence in Google, Ovid, PubMed, and UpToDate: results of a randomised trial , 2010, Postgraduate Medical Journal.

[10]  D. Sackett.  Evidence-Based Medicine: How to Practice and Teach EBM , 2018 .

[11]  Jimmy J. Lin,et al.  PubMed related articles: a probabilistic topic-based model for content similarity , 2007, BMC Bioinformatics.

[12]  R Brian Haynes,et al.  Filtering Medline for a clinical discipline: diagnostic test assessment framework , 2009, BMJ : British Medical Journal.

[13]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews , 2007, PLoS medicine.

[14]  Charles P. Friedman,et al.  Research Paper: Factors Associated with Success in Searching MEDLINE and Applying Evidence to Answer Clinical Questions , 2002, J. Am. Medical Informatics Assoc..

[15]  John W. Ely,et al.  Answering clinical questions in the ED. , 2008, The American journal of emergency medicine.

[16]  Fiona Simpson,et al.  Candidemia in critically ill patients: difference of outcome between medical and surgical patients , 2003, Intensive Care Medicine.

[17]  R. Haynes,et al.  Medline : analytical survey scientifically strong studies of diagnosis from Optimal search strategies for retrieving , 2004 .

[18]  Pertti Vakkari,et al.  Search effort degrades search output but improves task outcome , 2012, J. Assoc. Inf. Sci. Technol..

[19]  Douglas B. Fridsma,et al.  Research Paper: Effectiveness of Clinician-selected Electronic Information Resources for Answering Primary Care Physicians' Information Needs , 2006, J. Am. Medical Informatics Assoc..

[20]  D. Korenstein,et al.  Real-time EBM: From Bed Board to Keyboard and Back , 2007, Journal of General Internal Medicine.

[21]  Pieter de Vries Robbé,et al.  Analysis of queries sent to PubMed at the point of care: Observation of search behaviour in a medical teaching hospital , 2008, BMC Medical Informatics Decis. Mak..

[22]  Paul A. Fontelo,et al.  Utilization of the PICO framework to improve searching PubMed for clinical questions , 2007, BMC Medical Informatics Decis. Mak..

[23]  M. Green,et al.  Residents' medical information needs in clinic: are they being met? , 2000, The American journal of medicine.

[24]  K. A. McKibbon,et al.  Evaluating the impact of MEDLINE filters on evidence retrieval: study protocol , 2010, Implementation science : IS.

[25]  S. Satya‐Murti.  Evidence-based Medicine: How to Practice and Teach EBM , 1997 .

[26]  Ruth Gilbert,et al.  Evidence based medicine and the medical curriculum , 2008, BMJ : British Medical Journal.

[27]  H. Bastian,et al.  Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? , 2010, PLoS medicine.

[28]  Arjen Hoogendam,et al.  Answers to Questions Posed During Daily Patient Care Are More Likely to Be Answered by UpToDate Than PubMed , 2008, Journal of medical Internet research.

[29]  P. Poole,et al.  Mucolytic agents for chronic bronchitis or chronic obstructive pulmonary disease. , 2000, The Cochrane database of systematic reviews.

[30]  A. Garg,et al.  Diagnostic test systematic reviews: bibliographic search filters ("Clinical Queries") for diagnostic accuracy studies perform well. , 2009, Journal of clinical epidemiology.

[31]  J. Grimshaw,et al.  The Cochrane Collaboration 20 years in , 2013, Canadian Medical Association Journal.

[32]  David M. Rind,et al.  Association of a clinical knowledge support system with improved patient safety, reduced complications and shorter length of stay among Medicare beneficiaries in acute care hospitals in the United States , 2008, Int. J. Medical Informatics.

[33]  S. Straus,et al.  Evidence-based medicine: a commentary on common criticisms. , 2000, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[34]  Elmer V. Bernstam,et al.  A day in the life of PubMed: analysis of a typical day's query log. , 2007, Journal of the American Medical Informatics Association : JAMIA.

[35]  Sharon E Straus,et al.  Evidence-Based Medicine: How to Practice and Teach It , 2010 .

[36]  D. Moher,et al.  An alternative to the hand searching gold standard: validating methodological search filters using relative recall , 2006, BMC medical research methodology.

[37]  B. Manns,et al.  The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: An independent appraisal* , 2007, Critical care medicine.

[38]  Robert Brian Haynes,et al.  Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. , 2009, Health information and libraries journal.

[39]  Manesh R Patel,et al.  Randomized trial for answers to clinical questions: evaluating a pre-appraised versus a MEDLINE search protocol. , 2006, Journal of the Medical Library Association : JMLA.

[40]  W. Hersh,et al.  Factors associated with successful answering of clinical questions using an information retrieval system. , 2002, Bulletin of the Medical Library Association.

[41]  A. Webster,et al.  How to get the most from the medical literature: Searching the medical literature effectively , 2010, Nephrology.

[42]  Kristy Lundahl,et al.  Residents’ Patient-Specific Clinical Questions: Opportunities for Evidence-Based Learning , 2005, Academic medicine : journal of the Association of American Medical Colleges.