Using natural language processing to provide personalized learning opportunities from trainee clinical notes

OBJECTIVE Assessment of medical trainee learning through pre-defined competencies is now commonplace in schools of medicine. We describe a novel electronic advisor system using natural language processing (NLP) to identify two geriatric medicine competencies from medical student clinical notes in the electronic medical record: advance directives (AD) and altered mental status (AMS). MATERIALS AND METHODS Clinical notes from third year medical students were processed using a general-purpose NLP system to identify biomedical concepts and their section context. The system analyzed these notes for relevance to AD or AMS and generated custom email alerts to students with embedded supplemental learning material customized to their notes. Recall and precision of the two advisors were evaluated by physician review. Students were given pre and post multiple choice question tests broadly covering geriatrics. RESULTS Of 102 students approached, 66 students consented and enrolled. The system sent 393 email alerts to 54 students (82%), including 270 for AD and 123 for AMS. Precision was 100% for AD and 93% for AMS. Recall was 69% for AD and 100% for AMS. Students mentioned ADs for 43 patients, with all mentions occurring after first having received an AD reminder. Students accessed educational links 34 times from the 393 email alerts. There was no difference in pre (mean 62%) and post (mean 60%) test scores. CONCLUSIONS The system effectively identified two educational opportunities using NLP applied to clinical notes and demonstrated a small change in student behavior. Use of electronic advisors such as these may provide a scalable model to assess specific competency elements and deliver educational opportunities.

[1]  Matthew R. Thomas,et al.  Use of Electronic Medical Records by Physicians and Students in Academic Internal Medicine Settings , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[2]  J. Fisher,et al.  Opportunities and Challenges in Integrating Electronic Health Records Into Undergraduate Medical Education: A National Survey of Clerkship Directors , 2012, Teaching and learning in medicine.

[3]  R. Harden,et al.  Competency-based assessment: making it a reality. , 2003, Medical Teacher.

[4]  Elaine F Dannefer,et al.  The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. , 2007, Academic medicine : journal of the Association of American Medical Colleges.

[5]  R. Moss,et al.  Advance directives on admission. Clinical implications and analysis of the Patient Self-Determination Act of 1990. , 1991, JAMA.

[6]  D. Blumenthal,et al.  The "meaningful use" regulation for electronic health records. , 2010, The New England journal of medicine.

[7]  Yindalon Aphinyanagphongs,et al.  Text Categorization Models for Retrieval of High Quality Articles in Internal Medicine , 2003, AMIA.

[8]  Randolph A. Miller,et al.  Research Paper: Evaluation of a Method to Identify and Categorize Section Headers in Clinical Documents , 2009, J. Am. Medical Informatics Assoc..

[9]  Patricia A. Carney,et al.  An Analysis of Students' Clinical Experiences in an Integrated Primary Care Clerkship , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[10]  C. McDonald.  Protocol-based computer reminders, the quality of care and the non-perfectability of man. , 1976, The New England journal of medicine.

[11]  Randolph A. Miller,et al.  Research Paper: An Experiment Comparing Lexical and Statistical Methods for Extracting MeSH Terms from Clinical Free Text , 1998, J. Am. Medical Informatics Assoc..

[12]  D. Litzelman,et al.  The new formal competency-based curriculum and informal curriculum at Indiana University School of Medicine: overview and five-year analysis. , 2007, Academic medicine : journal of the Association of American Medical Colleges.

[13]  Timothy D. Imler,et al.  Clinical decision support with natural language processing facilitates determination of colonoscopy surveillance intervals. , 2014, Clinical gastroenterology and hepatology : the official clinical practice journal of the American Gastroenterological Association.

[14]  M. Whitcomb.  Redirecting the assessment of clinical competence. , 2007, Academic medicine : journal of the Association of American Medical Colleges.

[15]  J. Fisher,et al.  Medical Student Documentation in Electronic Health Records: A Collaborative Statement From the Alliance for Clinical Education , 2012, Teaching and learning in medicine.

[16]  Stephen R Smith,et al.  Assessing Students' Performances in a Competency‐based Curriculum , 2003, Academic medicine : journal of the Association of American Medical Colleges.

[17]  D. Reuben,et al.  Development and Validation of a Geriatric Knowledge Test for Medical Students , 2004, Journal of the American Geriatrics Society.

[18]  Joshua C. Denny,et al.  Tracking medical students' clinical experiences using natural language processing , 2009, J. Biomed. Informatics.

[19]  Reed G. Williams,et al.  Advancing resident assessment in graduate medical education. , 2009, Journal of graduate medical education.

[20]  R A Miller,et al.  Using POSTDOC to recognize biomedical concepts in medical school curricular documents. , 1994, Bulletin of the Medical Library Association.

[21]  A program to enhance competence in clinical transaction skills. , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[22]  D. Osterweil,et al.  Development and validation of a geriatrics knowledge test for primary care residents , 1997, Journal of General Internal Medicine.

[23]  M. Hojat,et al.  Documenting and comparing medical students' clinical experiences. , 2001, JAMA.

[24]  S B Bird,et al.  Emergency medicine resident patient care documentation using a hand-held computerized device. , 2001, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[25]  M. Langdorf,et al.  Quantification of procedures and resuscitations in an emergency medicine residency. , 1998, The Journal of emergency medicine.

[26]  Hua Xu,et al.  Data from clinical notes: a perspective on the tension between structure and flexible documentation , 2011, J. Am. Medical Informatics Assoc..

[27]  Ronald M. Harden,et al.  EditorialCompetency-based assessment: making it a reality , 2003 .

[28]  Betsy L. Humphreys,et al.  Technical Milestone: The Unified Medical Language System: An Informatics Research Collaboration , 1998, J. Am. Medical Informatics Assoc..

[29]  Randolph A. Miller,et al.  Development and Evaluation of a Clinical Note Section Header Terminology , 2008, AMIA.

[30]  William W. Stead,et al.  Research Paper: Effect of CPOE User Interface Design on User-Initiated Access to Educational and Patient Information during Clinical Care , 2005, J. Am. Medical Informatics Assoc..

[31]  Randolph A. Miller,et al.  Identifying QT prolongation from ECG impressions using a general-purpose Natural Language Processor , 2009, Int. J. Medical Informatics.

[32]  Joshua C. Denny,et al.  Automatic Capture of Student Notes to Augment Mentor Feedback and Student Performance on Patient Write-Ups , 2008, Journal of General Internal Medicine.

[33]  Hongfang Liu,et al.  Clinical decision support with automated text processing for cervical cancer screening , 2012, J. Am. Medical Informatics Assoc..

[34]  Alderson,et al.  Clinical experience of medical students in primary care: use of an electronic log in monitoring experience and in guiding education in the Cambridge Community Based Clinical Course , 1999, Medical education.

[36]  James J. Cimino,et al.  An "infobutton" for enabling patients to interpret on-line Pap smear reports , 2000, AMIA.

[37]  Anderson Spickard,et al.  Research Paper: "Understanding" Medical School Curriculum Content Using KnowledgeMap , 2003, J. Am. Medical Informatics Assoc..

[38]  R. Leipzig,et al.  Medicine in the 21st century: recommended essential geriatrics competencies for internal medicine and family medicine residents. , 2010, Journal of graduate medical education.

[39]  Joshua C. Denny,et al.  The KnowledgeMap Project: Development of a Concept-Based Medical School Curriculum Database , 2003, AMIA.

[40]  Keke Chen,et al.  Model Formulation: A Document Clustering and Ranking System for Exploring MEDLINE Citations , 2007, J. Am. Medical Informatics Assoc..

[41]  Lisa Mills,et al.  Personal Digital Assistants: A Review of Their Application in Graduate Medical Education , 2005, American journal of medical quality : the official journal of the American College of Medical Quality.

[42]  J. Fitzgerald,et al.  Brief report: Brief instrument to assess geriatrics knowledge of surgical and medical subspecialty house officers , 2006, Journal of General Internal Medicine.

[43]  I. Philibert,et al.  The next GME accreditation system--rationale and benefits. , 2012, The New England journal of medicine.

[44]  David W. Bates,et al.  Research Paper: KnowledgeLink: Impact of Context-Sensitive Information Retrieval on Clinicians' Information Needs , 2006, J. Am. Medical Informatics Assoc..

[45]  Gail M. Sullivan,et al.  Core Competencies for the Care of Older Patients: Recommendations of the American Geriatrics Society , 2000, Academic medicine : journal of the Association of American Medical Colleges.

[46]  Guilherme Del Fiol,et al.  Research Paper: Effectiveness of Topic-specific Infobuttons: A Randomized Controlled Trial , 2008, J. Am. Medical Informatics Assoc..