Combining Relevance Assignment with Quality of the Evidence to Support Guideline Development

Clinical practice guidelines are used to disseminate best practice to clinicians. Successful guidelines depend on literature that is both relevant to the questions posed and based on high quality research in accordance with evidence-based medicine. Meeting these standards requires extensive manual review. We describe a system that combines symbolic semantic processing with a statistical method for selecting both relevant and high quality studies. We focused on a cardiovascular risk factor guideline, and the overall performance of the system was 56% recall, 91% precision (F0.5-score 0.81). If quality of the evidence is not taken into account, performance drops to 62% recall, 79% precision (F0.5-score 0.75). We suggest that this system can potentially improve the efficiency of the literature review process in guideline development.

[1]  Marcelo Fiszman,et al.  Semantic Interpretation for the Biomedical Research Literature , 2005 .

[2]  Bruce E. Bray,et al.  Semantic Processing to Support Clinical Guideline Development , 2008, AMIA.

[3]  R. Brian Haynes,et al.  Developing Optimal Search Strategies for Detecting Clinically Relevant Qualitative Studies in MEDLINE , 2004, MedInfo.

[4]  R. Brian Haynes,et al.  Developing optimal search strategies for detecting clinically sound studies in MEDLINE. , 1994, Journal of the American Medical Informatics Association : JAMIA.

[5]  Yindalon Aphinyanagphongs,et al.  Research Paper: Text Categorization Models for High-Quality Article Retrieval in Internal Medicine , 2004, J. Am. Medical Informatics Assoc..

[6]  Aaron M. Cohen,et al.  Optimizing Feature Representation for Automated Systematic Review Work Prioritization , 2008, AMIA.

[7]  Graeme Hirst,et al.  Using Outcome Polarity in Sentence Extraction for Medical Question-Answering , 2006, AMIA.

[8]  Thomas C. Rindflesch,et al.  MedPost: a part-of-speech tagger for bioMedical text , 2004, Bioinform..

[9]  Jimmy J. Lin,et al.  Answering Clinical Questions with Knowledge-Based and Statistical Techniques , 2007, CL.

[10]  Halil Kilicoglu,et al.  Viewpoint Paper: Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence , 2009, J. Am. Medical Informatics Assoc..

[11]  Alan R. Aronson,et al.  Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program , 2001, AMIA.

[12]  Marcelo Fiszman,et al.  The interaction of domain knowledge and linguistic structure in natural language processing: interpreting hypernymic propositions in biomedical text , 2003, J. Biomed. Informatics.

[13]  S. Satya‐Murti.  Evidence-based Medicine: How to Practice and Teach EBM , 1997 .

[14]  Alastair Baker,et al.  Crossing the Quality Chasm: A New Health System for the 21st Century , 2001, BMJ : British Medical Journal.

[15]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..

[16]  M. Field,et al.  Clinical practice guidelines : directions for a new program , 1990 .

[17]  George Hripcsak,et al.  Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians , 2007, J. Biomed. Informatics.

[18]  Paul A. Fontelo,et al.  Utilization of the PICO framework to improve searching PubMed for clinical questions , 2007, BMC Medical Informatics Decis. Mak..

[19]  Jimmy J. Lin,et al.  Evaluation of PICO as a Knowledge Representation for Clinical Questions , 2006, AMIA.

[20]  Allen C. Browne,et al.  Lexical methods for managing variation in biomedical terminologies. , 1994, Proceedings. Symposium on Computer Applications in Medical Care.