Variationally regularized graph-based representation learning for electronic health records

Electronic Health Records (EHR) are high-dimensional data with implicit connections among thousands of medical concepts. These connections, for instance, the co-occurrence of diseases and lab-disease correlations can be informative when only a subset of these variables is documented by the clinician. A feasible approach to improving the representation learning of EHR data is to associate relevant medical concepts and utilize these connections. Existing medical ontologies can be the reference for EHR structures, but they place numerous constraints on the data source. Recent progress on graph neural networks (GNN) enables end-to-end learning of topological structures for non-grid or non-sequential data. However, there are problems to be addressed on how to learn the medical graph adaptively and how to understand the effect of medical graph on representation learning. In this paper, we propose a variationally regularized encoder-decoder graph network that achieves more robustness in graph structure learning by regularizing node representations. Our model outperforms the existing graph and non-graph based methods in various EHR predictive tasks based on both public data and real-world clinical data. Besides the improvements in empirical experiment performances, we provide an interpretation of the effect of variational regularization compared to standard graph neural network, using singular value analysis.

[1]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[2]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[3]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[4]  Yan Liu,et al.  Benchmarking deep learning models on large healthcare datasets , 2018, J. Biomed. Informatics.

[5]  Charles Elkan,et al.  Learning to Diagnose with LSTM Recurrent Neural Networks , 2015, ICLR.

[6]  Lei Zhang,et al.  Bridging the Gap between Spatial and Spectral Domains: A Survey on Graph Neural Networks , 2020, ArXiv.

[7]  Svetha Venkatesh,et al.  DeepCare: A Deep Dynamic Memory Model for Predictive Medicine , 2016, PAKDD.

[8]  Kazem Rahimi,et al.  BEHRT: Transformer for Electronic Health Records , 2019, Scientific Reports.

[9]  Soo Hyun Kwon,et al.  Diabetes and Alzheimer's Disease: Mechanisms and Nutritional Aspects , 2018, Clinical nutrition research.

[10]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[11]  N. Tangri,et al.  Predicting technique survival in peritoneal dialysis patients: comparing artificial neural networks and logistic regression , 2008, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[12]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[13]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[14]  Louis C. Tiao,et al.  Variational Spectral Graph Convolutional Networks , 2019, ArXiv.

[15]  Kelly Servick.  Another major drug candidate targeting the brain plaques of Alzheimer’s disease has failed. What’s left? , 2019, Science.

[16]  Andreas Spanias,et al.  Attend and Diagnose: Clinical Time Series Analysis using Attention Models , 2017, AAAI.

[17]  David Sontag,et al.  Learning Low-Dimensional Representations of Medical Concepts , 2016, CRI.

[18]  Bill Wilson,et al.  In a relationship , 2013 .

[19]  Aram Galstyan,et al.  Multitask learning and benchmarking with clinical time series data , 2017, Scientific Data.

[20]  Ping Zhang,et al.  Risk Prediction with Electronic Health Records: A Deep Learning Approach , 2016, SDM.

[21]  Tzu-chin Wu,et al.  Risk Factors for Development of Septic Shock in Patients with Urinary Tract Infection , 2015, BioMed research international.

[22]  Edward Choi,et al.  Graph Convolutional Transformer: Learning the Graphical Structure of Electronic Health Records , 2019, ArXiv.

[23]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[24]  Mohammad Taher Pilehvar,et al.  On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation , 2019, NGT@EMNLP-IJCNLP.

[25]  Xiaoning Qian,et al.  Semi-Implicit Graph Variational Auto-Encoders , 2019, NeurIPS.

[26]  Aykut Erdem.  FOR DEEP CONVOLUTIONAL NETWORKS , 2016 .

[27]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[28]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[29]  Le Song,et al.  GRAM: Graph-based Attention Model for Healthcare Representation Learning , 2016, KDD.

[30]  Jimeng Sun,et al.  Multi-layer Representation Learning for Medical Concepts , 2016, KDD.

[31]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[32]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[33]  David Sontag,et al.  Multi-task Prediction of Disease Onsets from Longitudinal Lab Tests , 2016, ArXiv.

[34]  David Sontag,et al.  Temporal Convolutional Neural Networks for Diagnosis from Lab Tests , 2015, ArXiv.

[35]  Hal Daumé,et al.  Deep Unordered Composition Rivals Syntactic Methods for Text Classification , 2015, ACL.

[36]  Paul Honeine,et al.  Bridging the Gap Between Spectral and Spatial Domains in Graph Neural Networks , 2020, ArXiv.

[37]  Dev Mehta,et al.  Why do trials for Alzheimer’s disease drugs keep failing? A discontinued drug perspective for 2010-2015 , 2017, Expert opinion on investigational drugs.

[38]  Alistair E. W. Johnson,et al.  The eICU Collaborative Research Database, a freely available multi-center database for critical care research , 2018, Scientific Data.

[39]  Joan Bruna,et al.  Deep Convolutional Networks on Graph-Structured Data , 2015, ArXiv.

[40]  Martin Jaggi,et al.  On the Relationship between Self-Attention and Convolutional Layers , 2019, ICLR.

[41]  Li Li,et al.  Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records , 2016, Scientific Reports.

[42]  Jimeng Sun,et al.  MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare , 2018, NeurIPS.

[43]  Ashish Vaswani,et al.  Stand-Alone Self-Attention in Vision Models , 2019, NeurIPS.

[44]  Walter F. Stewart,et al.  Doctor AI: Predicting Clinical Events via Recurrent Neural Networks , 2015, MLHC.

[45]  David Sontag,et al.  Multi-task Prediction of Disease Onsets from Longitudinal Laboratory Tests , 2016, MLHC.

[46]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[47]  Yujia Li,et al.  Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer , 2020, AAAI.

[48]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[49]  Ali Razavi,et al.  Preventing Posterior Collapse with delta-VAEs , 2019, ICLR.

[50]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[51]  Xiaoning Qian,et al.  Variational Graph Recurrent Neural Networks , 2019, NeurIPS.

[52]  Chung Y. Hsu,et al.  Septicemia is associated with increased risk for dementia: a population-based longitudinal study , 2017, Oncotarget.

[53]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[54]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[55]  Parisa Rashidi,et al.  Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis , 2017, IEEE Journal of Biomedical and Health Informatics.

[56]  Takaya Saito,et al.  The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets , 2015, PloS one.