Cohort selection for clinical trials: n2c2 2018 shared task track 1

OBJECTIVE Track 1 of the 2018 National NLP Clinical Challenges shared tasks focused on identifying which patients in a corpus of longitudinal medical records meet and do not meet identified selection criteria. MATERIALS AND METHODS To address this challenge, we annotated American English clinical narratives for 288 patients according to whether they met these criteria. We chose criteria from existing clinical trials that represented a variety of natural language processing tasks, including concept extraction, temporal reasoning, and inference. RESULTS A total of 47 teams participated in this shared task, with 224 participants in total. The participants represented 18 countries, and the teams submitted 109 total system outputs. The best-performing system achieved a micro F1 score of 0.91 using a rule-based approach. The top 10 teams used rule-based and hybrid systems to approach the problems. DISCUSSION Clinical narratives are open to interpretation, particularly in cases where the selection criterion may be underspecified. This leaves room for annotators to use domain knowledge and intuition in selecting patients, which may lead to error in system outputs. However, teams who consulted medical professionals while building their systems were more likely to have high recall for patients, which is preferable for patient selection systems. CONCLUSIONS There is not yet a 1-size-fits-all solution for natural language processing systems approaching this task. Future research in this area can look to examining criteria requiring even more complex inferences, temporal reasoning, and domain knowledge.

[1]  James Pustejovsky,et al.  A Methodology for Using Professional Knowledge in Corpus , 2013 .

[2]  Peter Szolovits,et al.  Challenges in Synthesizing Surrogate PHI in Narrative EMRs , 2015, Medical Data Privacy Handbook.

[3]  Long Chen,et al.  Clinical trial cohort selection based on multi-level rule-based natural language processing system , 2019, J. Am. Medical Informatics Assoc..

[4]  Grigorios Loukides,et al.  Medical Data Privacy Handbook , 2015, Springer International Publishing.

[5]  Sunghwan Sohn,et al.  Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications , 2010, J. Am. Medical Informatics Assoc..

[6]  C. Mann,et al.  Observational research methods. Research design II: cohort, cross sectional, and case-control studies , 2003, Emergency medicine journal : EMJ.

[7]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..

[8]  Michael Gertz,et al.  Multilingual and cross-domain temporal tagging , 2012, Language Resources and Evaluation.

[9]  Stuart J. Nelson,et al.  Normalized names for clinical drugs: RxNorm at 6 years , 2011, J. Am. Medical Informatics Assoc..

[10]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[11]  T. H. Kyaw,et al.  Multiparameter Intelligent Monitoring in Intensive Care II: A public-access intensive care unit database* , 2011, Critical care medicine.

[12]  Özlem Uzuner,et al.  Annotating risk factors for heart disease in clinical narratives for diabetic patients , 2015, J. Biomed. Informatics.

[13]  Marguerite Swietlik,et al.  Research Subject Enrollment by Primary Care Pediatricians Using an Electronic Health Record , 2007, AMIA.

[14]  Judith W. Dexheimer,et al.  Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department , 2014, J. Am. Medical Informatics Assoc..

[15]  S. Richardson,et al.  Adjusting for selection bias in retrospective, case-control studies. , 2008, Biostatistics.

[16]  Özlem Uzuner,et al.  Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus , 2015, J. Biomed. Informatics.

[17]  Daniel Fabbri,et al.  Development of an automated phenotyping algorithm for hepatorenal syndrome , 2018, J. Biomed. Informatics.

[18]  Chunhua Weng,et al.  Case Report: Electronic Screening Improves Efficiency in Clinical Trial Recruitment , 2009, J. Am. Medical Informatics Assoc..

[19]  Hongfang Liu,et al.  CLAMP – a toolkit for efficiently building customized clinical natural language processing pipelines , 2017, J. Am. Medical Informatics Assoc..

[20]  Peng Jin,et al.  Criteria2Query: a natural language interface to clinical databases for cohort definition , 2019, J. Am. Medical Informatics Assoc..

[21]  Peter J. Embi,et al.  Development of an Electronic Health Record-based Clinical Trial Alert System to Enhance Recruitment at the Point of Care , 2005, AMIA.

[22]  Vitaly Herasevich,et al.  The accuracy and efficiency of electronic screening for recruitment into a clinical trial on COPD. , 2011, Respiratory medicine.

[23]  Daniel J. Feller,et al.  Using Clinical Notes and Natural Language Processing for Automated HIV Risk Assessment , 2017, Journal of acquired immune deficiency syndromes.

[24]  Madia Essiet,et al.  Hybrid bag of approaches to characterize selection criteria for cohort identification , 2019, J. Am. Medical Informatics Assoc..

[25]  Louise Deléger,et al.  Increasing the efficiency of trial-patient matching: automated clinical trial eligibility Pre-screening for pediatric oncology patients , 2015, BMC Medical Informatics and Decision Making.

[26]  Ying Xiong,et al.  Cohort selection for clinical trials using hierarchical neural network , 2019, J. Am. Medical Informatics Assoc..

[27]  Judith W. Dexheimer,et al.  A Real-Time Automated Patient Screening System for Clinical Trials Eligibility in an Emergency Department: Design and Evaluation , 2019, JMIR medical informatics.

[28]  Markus Kreuzthaler,et al.  Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification , 2019, J. Am. Medical Informatics Assoc..

[29]  Alan R. Aronson,et al.  An overview of MetaMap: historical perspective and recent advances , 2010, J. Am. Medical Informatics Assoc..

[30]  S. T. Buckland,et al.  Computer-Intensive Methods for Testing Hypotheses. , 1990 .

[31]  William R. Hersh,et al.  Barriers to Retrieving Patient Information from Electronic Health Record Data: Failure Analysis from the TREC Medical Records Track , 2012, AMIA.

[32]  W. DuMouchel,et al.  Unlocking Clinical Data from Narrative Reports: A Study of Natural Language Processing , 1995, Annals of Internal Medicine.