A Comprehensive Evaluation of Multi-task Learning and Multi-task Pre-training on EHR Time-series Data

Multi-task learning (MTL) is a machine learning technique aiming to improve model performance by leveraging information across many tasks. It has been used extensively on various data modalities, including electronic health record (EHR) data. However, despite significant use on EHR data, there has been little systematic investigation of the utility of MTL across the diverse set of possible tasks and training schemes of interest in healthcare. In this work, we examine MTL across a battery of tasks on EHR time-series data. We find that while MTL does suffer from common negative transfer, we can realize significant gains via MTL pre-training combined with single-task fine-tuning. We demonstrate that these gains can be achieved in a task-independent manner and offer not only minor improvements under traditional learning, but also notable gains in a few-shot learning context, thereby suggesting this could be a scalable vehicle to offer improved performance in important healthcare contexts.

[1]  Quoc V. Le,et al.  Multi-task Sequence to Sequence Learning , 2015, ICLR.

[2]  David C. Kale,et al.  Do no harm: a roadmap for responsible machine learning for health care , 2019, Nature Medicine.

[3]  Rich Caruana,et al.  Multitask Learning: A Knowledge-Based Source of Inductive Bias , 1993, ICML.

[4]  Nigam H. Shah,et al.  The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data , 2019, PSB.

[5]  Sen Wu,et al.  Understanding and Improving Information Transfer in Multi-Task Learning , 2020, ICLR.

[6]  Dimitris Bertsimas,et al.  Predicting inpatient flow at a major hospital using interpretable analytics , 2020, medRxiv.

[7]  Rae Woong Park,et al.  Characterizing treatment pathways at scale using the OHDSI network , 2016, Proceedings of the National Academy of Sciences.

[8]  Jimeng Sun,et al.  Pre-training of Graph Augmented Transformers for Medication Recommendation , 2019, IJCAI.

[9]  Anna Goldenberg,et al.  Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks , 2019, MLHC.

[10]  Luca Oneto,et al.  Taking Advantage of Multitask Learning for Fair Classification , 2018, AIES.

[11]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[12]  Suchi Saria,et al.  Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Xiaodong Liu,et al.  Multi-Task Deep Neural Networks for Natural Language Understanding , 2019, ACL.

[14]  Andrew L. Beam,et al.  Practical guidance on artificial intelligence for health-care data. , 2019, The Lancet. Digital health.

[15]  Mihaela van der Schaar,et al.  Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes , 2017, NIPS.

[16]  A. Olry,et al.  Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database , 2019, European Journal of Human Genetics.

[17]  Harini Suresh,et al.  Learning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICU , 2018, KDD.

[18]  Qiang Yang,et al.  An Overview of Multi-task Learning , 2018 .

[19]  Aram Galstyan,et al.  Multitask learning and benchmarking with clinical time series data , 2017, Scientific Data.

[20]  Xavier Amatriain,et al.  Few-Shot Learning for Dermatological Disease Diagnosis , 2019, MLHC.

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  Jimeng Sun,et al.  Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review , 2018, J. Am. Medical Informatics Assoc..

[23]  Jon Kleinberg,et al.  Transfusion: Understanding Transfer Learning for Medical Imaging , 2019, NeurIPS.

[24]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[25]  Regina Barzilay,et al.  Investigating Resuscitation Code Assignment in the Intensive Care Unit using Structured and Unstructured Data. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[26]  Anthony Ndirango,et al.  Generalization in multitask deep neural classifiers: a statistical physics approach , 2019, NeurIPS.

[27]  V. N. Slee,et al.  The International Classification of Diseases: ninth revision (ICD-9) , 1978, Annals of internal medicine.

[28]  Mark Hasegawa-Johnson,et al.  Improved ASR for Under-Resourced Languages Through Multi-Task Learning with Acoustic Landmarks , 2018, INTERSPEECH.

[29]  Massimiliano Pontil,et al.  Multi-Task Feature Learning , 2006, NIPS.

[30]  Suchi Saria,et al.  A Framework for Individualizing Predictions of Disease Trajectories by Exploiting Multi-Resolution Structure , 2015, NIPS.

[31]  Yue Wang,et al.  Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? , 2020, ECCV.

[32]  Ming Zhang,et al.  Multi-task Learning via Adaptation to Similar Tasks for Mortality Prediction of Diverse Rare Diseases , 2020, ArXiv.

[33]  Marzyeh Ghassemi,et al.  MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III , 2019, CHIL.

[34]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[35]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[36]  Peter Szolovits,et al.  Understanding vasopressor intervention and weaning: risk prediction in a public heterogeneous clinical time series database , 2017, J. Am. Medical Informatics Assoc..

[37]  Anna Goldenberg,et al.  Bayesian Trees for Automated Cytometry Data Analysis. , 2019 .

[38]  Andrew Zisserman,et al.  Multi-task Self-Supervised Visual Learning , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[40]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[41]  Jorge Silva,et al.  Multi-Task Learning with Incomplete Data for Healthcare , 2018, ArXiv.

[42]  Sebastian Ruder,et al.  An Overview of Multi-Task Learning in Deep Neural Networks , 2017, ArXiv.

[43]  Brian W. Powers,et al.  Dissecting racial bias in an algorithm used to manage the health of populations , 2019, Science.

[44]  Kirk Roberts,et al.  Deep Patient Representation of Clinical Notes via Multi-Task Learning for Mortality Prediction. , 2019, AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science.

[45]  Yoshua Bengio,et al.  Professor Forcing: A New Algorithm for Training Recurrent Networks , 2016, NIPS.

[46]  Yu Zhang,et al.  A Survey on Multi-Task Learning , 2017, IEEE Transactions on Knowledge and Data Engineering.

[47]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[48]  Suchi Saria,et al.  Counterfactual Normalization: Proactively Addressing Dataset Shift Using Causal Mechanisms , 2018, UAI.

[49]  Vijay S. Pande,et al.  MoleculeNet: a benchmark for molecular machine learning , 2017, Chemical science.

[50]  S. Haneuse,et al.  Small Data Challenges of Studying Rare Diseases. , 2020, JAMA network open.

[51]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  S. Tamang,et al.  Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data , 2018, JAMA internal medicine.

[53]  Antitza Dantcheva,et al.  Mitigating Bias in Gender, Age and Ethnicity Classification: A Multi-task Convolution Neural Network Approach , 2018, ECCV Workshops.

[54]  Jeffrey Dean,et al.  Scalable and accurate deep learning with electronic health records , 2018, npj Digital Medicine.

[55]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[56]  George Hripcsak,et al.  Parameterizing time in electronic health record studies , 2015, J. Am. Medical Informatics Assoc..

[57]  Jimeng Sun,et al.  CONAN: Complementary Pattern Augmentation for Rare Disease Detection , 2019, AAAI.

[58]  Katherine A. Heller,et al.  An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection , 2017, MLHC.

[59]  Kazem Rahimi,et al.  BEHRT: Transformer for Electronic Health Records , 2019, Scientific Reports.

[60]  Anna Goldenberg,et al.  Dynamic Measurement Scheduling for Event Forecasting using Deep RL , 2019, ICML.

[61]  Andrew Slavin Ross,et al.  Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning , 2018, AMIA.

[62]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.