Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation

Clinical notes contain rich data, which is unexploited in predictive modeling compared to structured data. In this work, we developed a new text representation Clinical XLNet for clinical notes which also leverages the temporal information of the sequence of the notes. We evaluated our models on prolonged mechanical ventilation prediction problem and our experiments demonstrated that Clinical XLNet outperforms the best baselines consistently.

[1]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[2]  知秀 柴田.  5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding , 2020 .

[3]  A. Shorr,et al.  Prolonged acute mechanical ventilation, hospital resource utilization, and mortality in the United States , 2008, Critical care medicine.

[4]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[5]  Jyotishman Pathak,et al.  Using EHRs and Machine Learning for Heart Failure Survival Analysis , 2015, MedInfo.

[6]  Peter Szolovits,et al.  Towards the Creation of a Large Corpus of Synthetically-Identified Clinical Notes , 2018, ArXiv.

[7]  A. Rapsang,et al.  Scoring systems in the intensive care unit: A compendium , 2014, Indian journal of critical care medicine : peer-reviewed, official publication of Indian Society of Critical Care Medicine.

[8]  A. Bodenham,et al.  Tracheostomy in critically ill patients , 2010, European journal of anaesthesiology.

[9]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[10]  W. McGee,et al.  Expectations and outcomes of prolonged mechanical ventilation. , 2010, Critical care medicine.

[11]  J. Mancebo,et al.  Weaning from mechanical ventilation. , 1996, The European respiratory journal.

[12]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[13]  Brent Hadder,et al.  Validation and Extension of the Prolonged Mechanical Ventilation Prognostic Model (ProVent) Score for Predicting 1-Year Mortality after Prolonged Mechanical Ventilation. , 2015, Annals of the American Thoracic Society.

[14]  Sameer Hirji,et al.  Utility of 90-Day Mortality vs 30-Day Mortality as a Quality Metric for Transcatheter and Surgical Aortic Valve Replacement Outcomes. , 2019, JAMA cardiology.

[15]  Shannon S Carson,et al.  One-Year Trajectories of Care and Resource Utilization for Recipients of Prolonged Mechanical Ventilation , 2010, Annals of Internal Medicine.

[16]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[17]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[18]  J. L. Gall,et al.  APACHE II--a severity of disease classification system. , 1986, Critical care medicine.

[19]  S. Lemeshow,et al.  A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. , 1993, JAMA.

[20]  Yiming Yang,et al.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context , 2019, ACL.

[21]  Wei-Hung Weng,et al.  Publicly Available Clinical BERT Embeddings , 2019, Proceedings of the 2nd Clinical Natural Language Processing Workshop.

[22]  Rajesh Ranganath,et al.  ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission , 2019, ArXiv.

[23]  Yu Cheng,et al.  Exploiting Convolutional Neural Network for Risk Prediction with Medical Feature Embedding , 2017, ArXiv.

[24]  Gustavo Carneiro,et al.  Hidden stratification causes clinically meaningful failures in machine learning for medical imaging , 2019, CHIL.

[25]  Thomas Bice,et al.  To Trach or Not to Trach: Uncertainty in the Care of the Chronically Critically Ill , 2015, Seminars in Respiratory and Critical Care Medicine.

[26]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[27]  Lu Wang,et al.  Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation , 2018, KDD.

[28]  T. Murdoch,et al.  The inevitable application of big data to health care. , 2013, JAMA.

[29]  J. Vincent,et al.  The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure , 1996, Intensive Care Medicine.

[30]  Shannon S Carson,et al.  Increase in tracheostomy for prolonged mechanical ventilation in North Carolina, 1993–2002 , 2004, Critical care medicine.

[31]  W. Knaus,et al.  APACHE II: a severity of disease classification system. , 1985 .

[32]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[33]  B. Marsh,et al.  Weaning from mechanical ventilation , 2007, European Respiratory Journal.