Validation of a Crowdsourcing Methodology for Developing a Knowledge Base of Related Problem-Medication Pairs.

BACKGROUND Clinical knowledge bases of problem-medication pairs are necessary for many informatics solutions that improve patient safety, such as clinical summarization. However, developing these knowledge bases can be challenging. OBJECTIVE We sought to validate a previously developed crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large, non-university health care system with a widely used, commercially available electronic health record. METHODS We first retrieved medications and problems entered in the electronic health record by clinicians during routine care during a six month study period. Following the previously published approach, we calculated the link frequency and link ratio for each pair then identified a threshold cutoff for estimated problem-medication pair appropriateness through clinician review; problem-medication pairs meeting the threshold were included in the resulting knowledge base. We selected 50 medications and their gold standard indications to compare the resulting knowledge base to the pilot knowledge base developed previously and determine its recall and precision. RESULTS The resulting knowledge base contained 26,912 pairs, had a recall of 62.3% and a precision of 87.5%, and outperformed the pilot knowledge base containing 11,167 pairs from the previous study, which had a recall of 46.9% and a precision of 83.3%. CONCLUSIONS We validated the crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large non-university health care system with a widely used, commercially available electronic health record, indicating that the approach may be generalizable across healthcare settings and clinical systems. Further research is necessary to better evaluate the knowledge, to compare crowdsourcing with other approaches, and to evaluate if incorporating the knowledge into electronic health records improves patient outcomes.

[1]  A. Localio,et al.  Role of computerized physician order entry systems in facilitating medication errors. , 2005, JAMA.

[2]  Kathy L. MacLaughlin,et al.  Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening , 2013, Journal of the American Medical Informatics Association : JAMIA.

[3]  S D Small,et al.  Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. , 1995, JAMA.

[4]  R A Miller,et al.  Empirical Derivation of an Electronic Clinically Useful Problem Statement System , 1999, Annals of Internal Medicine.

[5]  Dean F Sittig,et al.  Improving Clinical Quality Indicators Through Electronic Health Records: It Takes More Than Just a Reminder , 2009, Pediatrics.

[6]  Adam Wright,et al.  Development of a Unified Computable Problem-Medication Knowledge base , 2014, AMIA.

[7]  Joel Carter,et al.  Meaningful Use. , 2016, Journal of palliative medicine.

[8]  D. Blumenthal,et al.  The "meaningful use" regulation for electronic health records. , 2010, The New England journal of medicine.

[9]  Mark Tuttle,et al.  Drug knowledge expressed as computable semantic triples. , 2011, Studies in health technology and informatics.

[10]  Peter L. Elkin,et al.  Initializing the VA medication reference terminology using UMLS metathesaurus co-occurrences , 2002, AMIA.

[11]  J. Giles.  Internet encyclopaedias go head to head , 2005, Nature.

[12]  Shannon Hughes,et al.  Can Online Consumers Contribute to Drug Knowledge? A Mixed-Methods Comparison of Consumer-Generated and Professionally Controlled Psychotropic Medication Information on the Internet , 2011, Journal of medical Internet research.

[13]  Dean F. Sittig,et al.  Clinical Summarization Capabilities of Commercially-available and Internally-developed Electronic Health Records , 2012, Applied Clinical Informatics.

[14]  Halil Kilicoglu,et al.  Semantic MEDLINE: A web application for managing the results of PubMed searches , 2008, SMBM 2008.

[15]  Jon D Duke,et al.  ADESSA: A Real-Time Decision Support Service for Delivery of Semantically Coded Adverse Drug Event Data. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[16]  Sean Ekins,et al.  Reaching Out to Collaborators: Crowdsourcing for Pharmaceutical Research , 2010, Pharmaceutical Research.

[17]  Richard A. Berk,et al.  An Introduction to Ensemble Methods for Data Analysis , 2004 .

[18]  Adam Wright,et al.  An automated technique for identifying associations between medications, laboratory results and problems , 2010, J. Biomed. Informatics.

[19]  William R. Hersh,et al.  Evaluation of biomedical text-mining systems: Lessons learned from information retrieval , 2005, Briefings Bioinform..

[20]  Dean F. Sittig,et al.  Legal, Ethical, and Financial Dilemmas in Electronic Health Record Adoption and Use , 2011, Pediatrics.

[21]  James J. Cimino,et al.  Research Paper: Providing Concept-oriented Views for Clinical Data Using a Knowledge-based System: An Evaluation , 2002, J. Am. Medical Informatics Assoc..

[22]  Benjamin M. Good,et al.  Crowdsourcing for bioinformatics , 2013, Bioinform..

[23]  Eric G. Poon,et al.  Research Paper: The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry , 2007, J. Am. Medical Informatics Assoc..

[24]  Tsung-Chun Tsai,et al.  Crowdsourcing techniques to create a fuzzy subset of SNOMED CT for semantic tagging of medical documents , 2010, International Conference on Fuzzy Systems.

[25]  Y. Han,et al.  Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System , 2005, Pediatrics.

[26]  Michelle Sweidan,et al.  Evaluation of features to support safety and quality in general practice clinical software , 2011, BMC Medical Informatics Decis. Mak..

[27]  Adam Wright,et al.  Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications , 2012, J. Am. Medical Informatics Assoc..

[28]  Adam Wright,et al.  Summarization of clinical information: A conceptual model , 2011, J. Biomed. Informatics.

[29]  Gilad J. Kuperman,et al.  Case Report: Comprehensive Analysis of a Medication Dosing Error Related to CPOE , 2005, J. Am. Medical Informatics Assoc..

[30]  Dean F. Sittig,et al.  Rights and responsibilities of users of electronic health records , 2012, Canadian Medical Association Journal.

[31]  George Hripcsak,et al.  Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study. , 2008, Journal of the American Medical Informatics Association : JAMIA.

[32]  Paul Wicks,et al.  The power of social networking in medicine , 2009, Nature Biotechnology.

[33]  Randolph A. Miller,et al.  Focus on health information technology, electronic health records and their financial impact: A framework for evaluating the appropriateness of clinical decision support alerts and responses , 2012, J. Am. Medical Informatics Assoc..

[34]  N. Laird,et al.  Incidence of Adverse Drug Events and Potential Adverse Drug Events: Implications for Prevention , 1995 .

[35]  Don Tapscott,et al.  Wikinomics: How Mass Collaboration Changes Everything , 2006 .

[36]  Marc Berg,et al.  Viewpoint Paper: Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors , 2003, J. Am. Medical Informatics Assoc..