Task 1: ShARe/CLEF eHealth Evaluation Lab 2013

This report outlines the Task 1 of the ShARe/CLEF eHealth evaluation lab pilot. This task focused on identification (1a) and normalization (1b) of diseases and disorders in clinical reports. It used annotations from the ShARe corpus. A total of 22 teams competed in Task 1a and 17 of them also participated Task 1b. The best systems had an F1 score of 0.75 (0.80 Precision, 0.71 Recall) in Task 1a and an accuracy of 0.59 in Task 1b. The organizers have made the text corpora, annotations, and evaluation tools available for future research and development.

[1]  Özlem Uzuner,et al.  Extracting medication information from clinical text , 2010, J. Am. Medical Informatics Assoc..

[2]  Anna Rumshisky,et al.  Evaluating temporal relations in clinical text: 2012 i2b2 Challenge , 2013, J. Am. Medical Informatics Assoc..

[3]  Peter Szolovits,et al.  Evaluating the state-of-the-art in automatic de-identification. , 2007, Journal of the American Medical Informatics Association : JAMIA.

[4]  Karen Spärck Jones.  Reflections on TREC , 1995, Inf. Process. Manag..

[5]  Jun'ichi Tsujii,et al.  GENIA corpus - a semantically annotated corpus for bio-textmining , 2003, ISMB.

[6]  Sanna Salanterä,et al.  Overview of the ShARe/CLEF eHealth Evaluation Lab 2013 , 2013, CLEF.

[7]  Ralph Grishman,et al.  Message Understanding Conference- 6: A Brief History , 1996, COLING.

[8]  Dina Demner-Fushman,et al.  Automatic segmentation of clinical texts , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  Serguei V. S. Pakhomov,et al.  Automated Disambiguation of Acronyms and Abbreviations in Clinical Texts: Window and Training Size Considerations , 2012, AMIA.

[10]  Angus Roberts,et al.  Building a semantically annotated corpus of clinical texts , 2009, J. Biomed. Informatics.

[11]  K. Bretonnel Cohen,et al.  A shared task involving multi-label classification of clinical free text , 2007, BioNLP@ACL.

[12]  Carol Friedman,et al.  A Study of Abbreviations in Clinical Notes , 2007, AMIA.

[13]  Gareth J. F. Jones,et al.  ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred Health Information Retrieval , 2014, CLEF.

[14]  John F. Hurdle,et al.  Extracting Information from Textual Documents in the Electronic Health Record: A Review of Recent Research , 2008, Yearbook of Medical Informatics.

[15]  Peter J. Haug,et al.  Comparing expert systems for identifying chest x-ray reports that support pneumonia , 1999, AMIA.

[16]  George Hripcsak,et al.  Technical Brief: Agreement, the F-Measure, and Reliability in Information Retrieval , 2005, J. Am. Medical Informatics Assoc..

[17]  Hua Xu,et al.  A comparative study of current clinical natural language processing systems on handling abbreviations in discharge summaries , 2012, AMIA.

[18]  Jun'ichi Tsujii,et al.  Corpus annotation for mining biomedical events from literature , 2008, BMC Bioinformatics.

[19]  Daniel A. Keim,et al.  Task 1 of the CLEF eHealth Evaluation Lab 2014 , 2014, CLEF.

[20]  Angus Roberts,et al.  The CLEF Corpus: Semantic Annotation of Clinical Text , 2007, AMIA.

[21]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[22]  Kirsten G. Engel,et al.  Patient understanding of emergency department discharge instructions: where are knowledge deficits greatest? , 2012, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[23]  William R. Hersh,et al.  Enhancing Access to the Bibliome: The TREC Genomics Track , 2004, MedInfo.

[24]  Fei Xia,et al.  Community annotation experiment for ground truth generation for the i2b2 medication challenge , 2010, J. Am. Medical Informatics Assoc..

[25]  Chen-Tan Lin,et al.  Review Paper: The Effects of Promoting Patient Access to Medical Records: A Review , 2003, J. Am. Medical Informatics Assoc..

[26]  George Hripcsak,et al.  Evaluation of training with an annotation schema for manual annotation of clinical conditions from emergency department reports , 2008, Int. J. Medical Informatics.

[27]  Wendy W. Chapman,et al.  A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries , 2001, J. Biomed. Informatics.

[28]  Olivier Bodenreider,et al.  Exploring semantic groups through visual approaches , 2003, J. Biomed. Informatics.

[29]  Danielle L. Mowery,et al.  Task 2 : ShARe/CLEF eHealth Evaluation Lab 2014 , 2013 .

[30]  Christopher G. Chute,et al.  Technical Brief: Mayo Clinic NLP System for Patient Smoking Status Identification , 2008, J. Am. Medical Informatics Assoc..

[31]  Danielle L. Mowery,et al.  Semantic Annotation of Clinical Events for Generating a Problem List , 2013, AMIA.

[32]  Yuan Luo,et al.  Identifying patient smoking status from medical discharge records. , 2008, Journal of the American Medical Informatics Association : JAMIA.

[33]  Alexander S. Yeh,et al.  More accurate tests for the statistical significance of result differences , 2000, COLING.

[34]  Shuying Shen,et al.  A Prototype Tool Set to Support Machine-Assisted Annotation , 2012, BioNLP@HLT-NAACL.

[35]  Christopher G. Chute,et al.  Word sense disambiguation across two domains: Biomedical literature and clinical notes , 2008, J. Biomed. Informatics.

[36]  R G Mark,et al.  MIMIC II: a massive temporal ICU patient database to support research in intelligent patient monitoring , 2002, Computers in Cardiology.

[37]  Edward H. Shortliffe,et al.  Viewpoint: The Unified Medical Language System: Toward a Collaborative Approach for Solving Terminologic Problems , 1998, J. Am. Medical Informatics Assoc..

[38]  Shuying Shen,et al.  2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text , 2011, J. Am. Medical Informatics Assoc..