Randomized controlled trial of an automated problem list with improved sensitivity

PURPOSE To improve the completeness and timeliness of an electronic problem list, we have developed a system using Natural Language Processing (NLP) to automatically extract potential medical problems from clinical, free-text documents; these problems are then proposed for inclusion in an electronic problem list management application. METHODS A prospective randomized controlled evaluation of the Automatic Problem List (APL) system in an intensive care unit and in a cardiovascular surgery unit is reported here. A total of 247 patients were enrolled: 76 in an initial control phase and 171 in the randomized controlled trial that followed. During this latter phase, patients were randomly assigned to a control or an intervention group. All patients had their documents analyzed by the system, but the medical problems discovered were only proposed in the problem list for intervention patients. We measured the sensitivity, specificity, positive and negative predictive values, likelihood ratios and the timeliness of the problem lists. RESULTS Our system significantly increased the sensitivity of the problem lists in the intensive care unit, from about 9% to 41%, and even 77% if problems automatically proposed but not acknowledged by users were also considered. Timeliness of addition of problems to the list was greatly improved, with a time between a problem's first mention in a clinical document and its addition to the problem list reduced from about 6 days to less than 2 days. No significant effect was observed in the cardiovascular surgery unit.

[1]  Steven H. Brown,et al.  Development of a Structured Problem-List Management System at Vanderbilt , 1998, AMIA.

[2]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[3]  T H Payne,et al.  How useful is the UMLS metathesaurus in developing a controlled vocabulary for an automated problem list? , 1993, Proceedings. Symposium on Computer Applications in Medical Care.

[4]  Astrid M. van Ginneken,et al.  The computerized patient record: balancing effort and benefit , 2002, Int. J. Medical Informatics.

[5]  Paul D. Clayton,et al.  Using LOINC to link an EMR to the pertinent paragraph in a structured reference knowledge base , 2002, AMIA.

[6]  Samson W. Tu,et al.  The helpful patient record system: problem oriented and knowledge based , 2002, AMIA.

[7]  James R. Campbell,et al.  Just-in-time coding of the problem list in a clinical environment , 1998, AMIA.

[8]  B Starfield,et al.  Concordance Between Medical Records and Observations Regarding Information on Coordination of Care , 1979, Medical care.

[9]  E. B. Steen,et al.  The Computer-Based Patient Record: An Essential Technology for Health Care , 1992, Annals of Internal Medicine.

[10]  D W Simborg,et al.  Information Factors Affecting Problem Follow-Up in Ambulatory Care , 1976, Medical care.

[11]  A. W. Pratt.  Medicine, Computers, and Linguistics , 1973 .

[12]  Donaldson Ms,et al.  Improving the master problem list: a case study in changing clinician behavior. , 1985 .

[13]  Peter J. Haug,et al.  Natural language processing to extract medical problems from electronic clinical documents: Performance evaluation , 2006, J. Biomed. Informatics.

[14]  James R. Campbell.  Strategies for problem list implementation in a complex clinical enterprise , 1998, AMIA.

[15]  C M Ashton,et al.  An empirical assessment of the validity of explicit and implicit process-of-care criteria for quality assessment. , 1999, Medical care.

[16]  J Zelingher,et al.  Categorization of free-text problem lists: an effective method of capturing clinical data. , 1995, Proceedings. Symposium on Computer Applications in Medical Care.

[17]  Susanne M. Humphrey,et al.  The NLM Indexing Initiative's Medical Text Indexer , 2004, MedInfo.

[18]  David W. Bates,et al.  Automated coded ambulatory problem lists: evaluation of a vocabulary and a data entry tool , 2003, Int. J. Medical Informatics.

[19]  Charles Safran,et al.  An Evaluation of UMLS as a Controlled Terminology for the Problem List Toolkit , 1998, MedInfo.

[20]  Alan R. Aronson,et al.  Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program , 2001, AMIA.

[21]  L. Weed.  Medical records that guide and teach. , 1968, The New England journal of medicine.

[22]  John G. Gray,et al.  Medicine , 1902, Glasgow Medical Journal.

[23]  M S Donaldson,et al.  Improving the master problem list: a case study in changing clinician behavior. , 1985, QRB. Quality review bulletin.

[24]  H. Scherpbier,et al.  A simple approach to physician entry of patient problem list. , 1994, Proceedings. Symposium on Computer Applications in Medical Care.

[25]  Peter J. Haug,et al.  Bmc Medical Informatics and Decision Making Automation of a Problem List Using Natural Language Processing , 2005 .

[26]  Mary K Goldstein,et al.  Accuracy of computerized outpatient diagnoses in a Veterans Affairs general medicine clinic. , 2002, The American journal of managed care.

[27]  Christopher G. Chute,et al.  Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System , 1997, AMIA.

[28]  James R. Campbell,et al.  A comparison of four schemes for codification of problem lists. , 1994, Proceedings. Symposium on Computer Applications in Medical Care.

[29]  S. Huff,et al.  Building a Comprehensive Clinical Information System from Components , 2003, Methods of Information in Medicine.

[30]  Peter Spyns.  Natural Language Processing in Medicine: An Overview , 1996, Methods of Information in Medicine.

[31]  Jerome Wang,et al.  An Applied Evaluation of SNOMED CT as a Clinical Vocabulary for the Computerized Diagnosis and Problem List , 2003, AMIA.

[32]  Denise R. Aberle,et al.  Automated Medical Problem List Generation: Towards a Patient TimeLine , 2004, MedInfo.

[33]  L. Kohn,et al.  To Err Is Human : Building a Safer Health System , 2007 .

[34]  Katherine Schoeffler,et al.  Extracting medical knowledge for a coded problem list vocabulary from the UMLS Knowledge Sources , 1998, AMIA.