The Impact of Automatic Pre-annotation in Clinical Note Data Element Extraction - the CLEAN Tool

Author(s): Kuo, Tsung-Ting; Huh, Jina; Kim, Jihoon; El-Kareh, Robert; Singh, Siddharth; Feupe, Stephanie Feudjio; Kuri, Vincent; Lin, Gordon; Day, Michele E; Ohno-Machado, Lucila; Hsu, Chun-Nan | Abstract: Objective. Annotation is expensive but essential for clinical note review and clinical natural language processing (cNLP). However, the extent to which computer-generated pre-annotation is beneficial to human annotation is still an open question. Our study introduces CLEAN (CLinical note rEview and ANnotation), a pre-annotation-based cNLP annotation system to improve clinical note annotation of data elements, and comprehensively compares CLEAN with the widely-used annotation system Brat Rapid Annotation Tool (BRAT). Materials and Methods. CLEAN includes an ensemble pipeline (CLEAN-EP) with a newly developed annotation tool (CLEAN-AT). A domain expert and a novice user/annotator participated in a comparative usability test by tagging 87 data elements related to Congestive Heart Failure (CHF) and Kawasaki Disease (KD) cohorts in 84 public notes. Results. CLEAN achieved higher note-level F1-score (0.896) over BRAT (0.820), with significant difference in correctness (P-value l 0.001), and the mostly related factor being system/software (P-value l 0.001). No significant difference (P-value 0.188) in annotation time was observed between CLEAN (7.262 minutes/note) and BRAT (8.286 minutes/note). The difference was mostly associated with note length (P-value l 0.001) and system/software (P-value 0.013). The expert reported CLEAN to be useful/satisfactory, while the novice reported slight improvements. Discussion. CLEAN improves the correctness of annotation and increases usefulness/satisfaction with the same level of efficiency. Limitations include untested impact of pre-annotation correctness rate, small sample size, small user size, and restrictedly validated gold standard. Conclusion. CLEAN with pre-annotation can be beneficial for an expert to deal with complex annotation tasks involving numerous and diverse target data elements.

[1]  Andreas Holzinger,et al.  Usability engineering methods for software developers , 2005, CACM.

[2]  Louise Deléger,et al.  Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements , 2013, J. Am. Medical Informatics Assoc..

[3]  Christopher G. Chute,et al.  Constructing Evaluation Corpora for Automated Clinical Named Entity Recognition , 2008, LREC.

[4]  Özlem Uzuner,et al.  Viewpoint Paper: Recognizing Obesity and Comorbidities in Sparse Data , 2009, J. Am. Medical Informatics Assoc..

[5]  Jonathan R. Nebeker,et al.  reducing the Manual Burden of Medical Record Review through Informatics : 772 , 2014 .

[6]  Olga Patterson,et al.  Check it with Chex: A Validation Tool for Iterative NLP Development , 2014, AMIA.

[7]  Shuying Shen,et al.  A Prototype Tool Set to Support Machine-Assisted Annotation , 2012, BioNLP@HLT-NAACL.

[8]  Erik M. van Mulligen,et al.  Using an ensemble system to improve concept extraction from clinical records , 2012, J. Biomed. Informatics.

[9]  Wei Ma,et al.  RxNorm: prescription for electronic drug information exchange , 2005, IT Professional.

[10]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[11]  Wendy W. Chapman,et al.  Anaphoric relations in the clinical narrative: corpus creation , 2011, J. Am. Medical Informatics Assoc..

[12]  Juan D. Chaparro,et al.  Building a Natural Language Processing Tool to Identify Patients With High Clinical Suspicion for Kawasaki Disease from Emergency Department Notes. , 2016, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[13]  Özlem Uzuner,et al.  Extracting medication information from clinical text , 2010, J. Am. Medical Informatics Assoc..

[14]  C. McDonald,et al.  LOINC, a universal standard for identifying laboratory observations: a 5-year update. , 2003, Clinical chemistry.

[15]  Sidney L. Smith,et al.  Guidelines for Designing User Interface Software , 1986 .

[16]  Özlem Uzuner,et al.  Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus , 2015, J. Biomed. Informatics.

[17]  Son Doan,et al.  Recognition of medication information from discharge summaries using ensembles of classifiers , 2012, BMC Medical Informatics and Decision Making.

[18]  Shuying Shen,et al.  Automated extraction of ejection fraction for quality measurement using regular expressions in Unstructured Information Management Architecture (UIMA) for heart failure , 2012, J. Am. Medical Informatics Assoc..

[19]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[20]  Kent A. Spackman,et al.  SNOMED clinical terms: overview of the development process and project status , 2001, AMIA.

[21]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[22]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[23]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[24]  Hua Xu,et al.  Identifying risk factors for heart disease over time: Overview of 2014 i2b2/UTHealth shared task Track 2 , 2015, J. Biomed. Informatics.

[25]  Sunghwan Sohn,et al.  Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications , 2010, J. Am. Medical Informatics Assoc..

[26]  Hongfang Liu,et al.  Using machine learning for concept extraction on clinical documents from multiple data sources , 2011, J. Am. Medical Informatics Assoc..

[27]  Lucila Ohno-Machado,et al.  pSCANNER: patient-centered Scalable National Network for Effectiveness Research , 2014, J. Am. Medical Informatics Assoc..

[28]  Urmila Kukreja,et al.  RUI: Recording user input from interfaces under Windows and Mac OS X , 2006, Behavior research methods.

[29]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[30]  Fei Xia,et al.  Community annotation experiment for ground truth generation for the i2b2 medication challenge , 2010, J. Am. Medical Informatics Assoc..

[31]  Wilbert O. Galitz,et al.  The Essential Guide to User Interface Design: An Introduction to GUI Design Principles and Techniques , 1996 .

[32]  Xin Liu,et al.  An automatic system to identify heart disease risk factors in clinical texts over time , 2015, J. Biomed. Informatics.

[33]  Olga Patterson,et al.  Extraction of Vital Signs from Clinical Notes , 2015, MedInfo.

[34]  Yuan Luo,et al.  Identifying patient smoking status from medical discharge records. , 2008, Journal of the American Medical Informatics Association : JAMIA.

[35]  Frank E. Ritter,et al.  A Design, Tests and Considerations for Improving Keystroke and Mouse Loggers , 2013, Interact. Comput..

[36]  Son Doan,et al.  Ensembles of NLP Tools for Data Element Extraction from Clinical Notes , 2016, AMIA.

[37]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..

[38]  Peter Szolovits,et al.  Evaluating the state-of-the-art in automatic de-identification. , 2007, Journal of the American Medical Informatics Association : JAMIA.

[39]  Alan R. Aronson,et al.  Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program , 2001, AMIA.

[40]  Andrea Esuli,et al.  An enhanced CRFs-based system for information extraction from radiology reports , 2013, J. Biomed. Informatics.

[41]  Hua Xu,et al.  A study of active learning methods for named entity recognition in clinical text , 2015, J. Biomed. Informatics.

[42]  Özlem Uzuner,et al.  Annotating risk factors for heart disease in clinical narratives for diabetic patients , 2015, J. Biomed. Informatics.

[43]  Sampo Pyysalo,et al.  brat: a Web-based Tool for NLP-Assisted Text Annotation , 2012, EACL.

[44]  Anita Burgun-Parenthoine,et al.  Reviewing 741 patients records in two hours with FASTVISU , 2015, AMIA.

[45]  Shuying Shen,et al.  Evaluating the state of the art in coreference resolution for electronic medical records , 2012, J. Am. Medical Informatics Assoc..

[46]  Cyril Grouin,et al.  De-identification of clinical notes in French: towards a protocol for reference corpus development , 2014, J. Biomed. Informatics.

[47]  Alan R. Aronson,et al.  An overview of MetaMap: historical perspective and recent advances , 2010, J. Am. Medical Informatics Assoc..

[48]  Özlem Uzuner,et al.  Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1 , 2015, J. Biomed. Informatics.

[49]  Anna Rumshisky,et al.  Evaluating temporal relations in clinical text: 2012 i2b2 Challenge , 2013, J. Am. Medical Informatics Assoc..

[50]  Sanna Salanterä,et al.  Overview of the ShARe/CLEF eHealth Evaluation Lab 2013 , 2013, CLEF.

[51]  Shuying Shen,et al.  2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text , 2011, J. Am. Medical Informatics Assoc..

[52]  Alexa T. McCray,et al.  Modeling the Autism Spectrum Disorder Phenotype , 2013, Neuroinformatics.

[53]  Stéphane M. Meystre,et al.  Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text , 2014, J. Biomed. Informatics.

[54]  Hyeon-Eui Kim,et al.  Review and Evaluation of the State of Standardization of Computable Phenotype , 2016, AMIA.

[55]  Jimeng Sun,et al.  Automatic identification of heart failure diagnostic criteria, using text analysis of clinical notes from electronic health records , 2014, Int. J. Medical Informatics.

[56]  Son Doan,et al.  Application of information technology: MedEx: a medication information extraction system for clinical narratives , 2010, J. Am. Medical Informatics Assoc..

[57]  Wendy W. Chapman,et al.  A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries , 2001, J. Biomed. Informatics.

[58]  Ellen Riloff,et al.  Stacked Generalization for Medical Concept Extraction from Clinical Notes , 2015, BioNLP@IJCNLP.