Diagnostic Inferencing via Improving Clinical Concept Extraction with Deep Reinforcement Learning: A Preliminary Study

Clinical diagnostic inferencing is a complex task, which often requires significant medical research and investigation based on an underlying clinical scenario. This paper proposes a novel approach by formulating the task as a reinforcement learning problem such that the system can infer the most probable diagnoses by optimizing clinical concept extraction from a free text case narrative via leveraging relevant external evidence. Such a formulation is deemed to be suitable due to the inherent complexity of the task and unavailability of sufficient annotated data. During training, the agent tries to learn the optimal policy through iterative search and consolidation of the most relevant clinical concepts that best describe a correct diagnosis. A deep Q-network architecture is trained to optimize a reward function that measures the accuracy of the candidate diagnoses and clinical concepts. Our preliminary experiments on the TREC CDS dataset demonstrate the effectiveness of our system over non-reinforcement learning-based strong baselines.

[1]  Ellen Riloff,et al.  A Study of Concept Extraction Across Different Types of Clinical Notes , 2015, AMIA.

[2]  Lee Brooks,et al.  Non‐analytical models of clinical reasoning: the role of experience , 2007, Medical education.

[3]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[4]  Radhika Poolla,et al.  A reinforcement learning approach to obtain treatment strategies in sequential medical decision problems , 2003 .

[5]  S. Mamede,et al.  Cognitive diagnostic error in internal medicine. , 2013, European journal of internal medicine.

[6]  Sanda M. Harabagiu,et al.  Medical Question Answering for Clinical Decision Support , 2016, CIKM.

[7]  Erica Bell,et al.  Evidence in clinical reasoning: a computational linguistics analysis of 789,712 medical case summaries 1983–2012 , 2015, BMC Medical Informatics and Decision Making.

[8]  Gary C. Borchardt,et al.  External Knowledge Sources for Question Answering , 2005, TREC.

[9]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[10]  Xuchen Yao,et al.  Information Extraction over Structured Data: Question Answering with Freebase , 2014, ACL.

[11]  Siddhartha Jonnalagadda,et al.  Enhancing clinical concept extraction with distributional semantics , 2012, J. Biomed. Informatics.

[12]  Charles Elkan,et al.  Learning to Diagnose with LSTM Recurrent Neural Networks , 2015, ICLR.

[13]  Oladimeji Farri,et al.  Condensed Memory Networks for Clinical Diagnostic Inferencing , 2016, AAAI.

[14]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[15]  Danqi Chen,et al.  Reasoning With Neural Tensor Networks for Knowledge Base Completion , 2013, NIPS.

[16]  Bernard Charlin,et al.  An analysis of clinical reasoning through a recent and comprehensive approach: the dual-process theory , 2011, Medical education online.

[17]  Alexander Kotov,et al.  Optimization Method for Weighting Explicit and Latent Concepts in Clinical Decision Support Queries , 2016, ICTIR.

[18]  Ming Zhou,et al.  Question Answering over Freebase with Multi-Column Convolutional Neural Networks , 2015, ACL.

[19]  Daniel S. Weld,et al.  Open Information Extraction Using Wikipedia , 2010, ACL.

[20]  Mathieu Nendaz,et al.  Diagnostic errors and flaws in clinical reasoning: mechanisms and prevention in practice. , 2012, Swiss medical weekly.

[21]  Ellen M. Voorhees,et al.  Overview of the TREC 2014 Clinical Decision Support Track , 2014, TREC.

[22]  Qin Zhang,et al.  Clinical diagnosis expert system based on dynamic uncertain causality graph , 2014, 2014 IEEE 7th Joint International Information Technology and Artificial Intelligence Conference.

[23]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[24]  Walter F. Stewart,et al.  Doctor AI: Predicting Clinical Events via Recurrent Neural Networks , 2015, MLHC.

[25]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..

[26]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[27]  Regina Barzilay,et al.  Language Understanding for Text-based Games using Deep Reinforcement Learning , 2015, EMNLP.

[28]  Jimeng Sun,et al.  SympGraph: a framework for mining clinical notes through symptom relation graphs , 2012, KDD.

[29]  André Kushniruk,et al.  Analysis of Complex Decision-Making Processes in Health Care: Cognitive Approaches to Health Informatics , 2001, J. Biomed. Informatics.

[30]  Regina Barzilay,et al.  Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning , 2016, EMNLP.

[31]  Massimo Piccardi,et al.  Bidirectional LSTM-CRF for Clinical Concept Extraction , 2016, ClinicalNLP@COLING 2016.

[32]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[33]  Siddharth Patwardhan,et al.  WatsonPaths: Scenario-Based Question Answering and Inference over Unstructured Information , 2017, AI Mag..

[34]  Jason Weston,et al.  Key-Value Memory Networks for Directly Reading Documents , 2016, EMNLP.

[35]  Yuan Ling,et al.  Improving Clinical Diagnosis Inference through Integration of Structured and Unstructured Knowledge , 2017 .

[36]  Nancy F. Lenfestey,et al.  Cognitive interventions to reduce diagnostic error: a narrative review , 2012, BMJ quality & safety.

[37]  Tiejun Zhao,et al.  Knowledge-Based Question Answering as Machine Translation , 2014, ACL.

[38]  Xiaojun Wan,et al.  Graph-Based Multi-Modality Learning for Clinical Decision Support , 2016, CIKM.

[39]  Gang Pan,et al.  Semantic Health Knowledge Graph: Semantic Integration of Heterogeneous Medical Knowledge and Services , 2017, BioMed research international.

[40]  Jason Weston,et al.  Question Answering with Subgraph Embeddings , 2014, EMNLP.

[41]  Rahul Gupta,et al.  Knowledge base completion via search-based question answering , 2014, WWW.

[42]  M. Kosorok,et al.  Reinforcement Learning Strategies for Clinical Trials in Nonsmall Cell Lung Cancer , 2011, Biometrics.

[43]  R. Bellman.  A Markovian Decision Process , 1957 .

[44]  Anselmo Cardoso de Paiva,et al.  Application on Reinforcement Learning for Diagnosis Based on Medical Image , 2008 .

[45]  Jennifer S Myers,et al.  Teaching about how doctors think: a longitudinal curriculum in cognitive bias and diagnostic error for residents , 2013, BMJ quality & safety.

[46]  Ellen M. Voorhees,et al.  State-of-the-art in biomedical literature retrieval for clinical cases: a survey of the TREC 2014 CDS track , 2016, Information Retrieval Journal.

[47]  Erik T. Mueller,et al.  Watson: Beyond Jeopardy! , 2013, Artif. Intell..

[48]  Jimeng Sun,et al.  RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism , 2016, NIPS.

[49]  G H Guyatt,et al.  Users' guides to the medical literature: XV. How to use an article about disease probability for differential diagnosis. Evidence-Based Medicine Working Group. , 1999, JAMA.

[50]  Joelle Pineau,et al.  Informing sequential clinical decision-making through reinforcement learning: an empirical study , 2010, Machine Learning.