Missingness as Stability: Understanding the Structure of Missingness in Longitudinal EHR data and its Impact on Reinforcement Learning in Healthcare

There is an emerging trend in the reinforcement learning for healthcare literature. In order to prepare longitudinal, irregularly sampled, clinical datasets for reinforcement learning algorithms, many researchers will resample the time series data to short, regular intervals and use last-observation-carried-forward (LOCF) imputation to fill in these gaps. Typically, they will not maintain any explicit information about which values were imputed. In this work, we (1) call attention to this practice and discuss its potential implications; (2) propose an alternative representation of the patient state that addresses some of these issues; and (3) demonstrate in a novel but representative clinical dataset that our alternative representation yields consistently better results for achieving optimal control, as measured by off-policy policy evaluation, compared to representations that do not incorporate missingness information.

[1]  M Greaves,et al.  Guidelines on the use and monitoring of heparin , 2006, British journal of haematology.

[2]  Hong Yu,et al.  Towards High Confidence Off-Policy Reinforcement Learning for Clinical Applications , 2018 .

[3]  Shamim Nemati,et al.  Does the "Artificial Intelligence Clinician" learn optimal treatment strategies for sepsis in intensive care? , 2019, ArXiv.

[4]  Osman Y. Özaltın,et al.  The value of missing information in severity of illness score development , 2019, J. Biomed. Informatics.

[5]  Andrew Slavin Ross,et al.  Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning , 2018, AMIA.

[6]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[7]  Barbara E. Engelhardt,et al.  A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units , 2017, UAI.

[8]  Jiming Liu,et al.  Reinforcement Learning in Healthcare: A Survey , 2019, ACM Comput. Surv..

[9]  Shamim Nemati,et al.  Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[10]  Fredrik D. Johansson,et al.  Guidelines for reinforcement learning in healthcare , 2019, Nature Medicine.

[11]  Susan Regan,et al.  Challenges to the effective use of unfractionated heparin in the hospitalized management of acute thrombosis. , 2003, Archives of internal medicine.

[12]  David C. Kale,et al.  Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series , 2016, MLHC.

[13]  Tadayoshi Fushiki,et al.  Estimation of prediction error by using K-fold cross-validation , 2011, Stat. Comput..

[14]  Mark Hudson,et al.  Guidelines on the management of abnormal liver blood tests , 2017, Gut.

[15]  T. Vondracek,et al.  Antifactor Xa Levels versus Activated Partial Thromboplastin Time for Monitoring Unfractionated Heparin , 2012, Pharmacotherapy.

[16]  O. Miettinen,et al.  Confounding and effect-modification. , 1974, American journal of epidemiology.

[17]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[18]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[19]  S. Saria.  Individualized sepsis treatment using reinforcement learning , 2018, Nature Medicine.

[20]  Mit Critical Data.  Secondary Analysis of Electronic Health Records , 2016 .

[21]  Brett K. Beaulieu-Jones,et al.  Characterizing and Managing Missing Structured Data in Electronic Health Records: Data Analysis , 2017, bioRxiv.

[22]  M. Brigden.  Clinical utility of the erythrocyte sedimentation rate. , 1999, American family physician.

[23]  Martin Tran,et al.  The frequency of testing for glycated haemoglobin, HbA1c, is linked to the probability of achieving target levels in patients with suboptimally controlled diabetes mellitus , 2018, Clinical chemistry and laboratory medicine.

[24]  Philip S. Thomas,et al.  High-Confidence Off-Policy Evaluation , 2015, AAAI.

[25]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[26]  S. F. Buck.  A Method of Estimation of Missing Values in Multivariate Data Suitable for Use with an Electronic Computer , 1960 .

[27]  Susan C. Weber,et al.  STRIDE - An Integrated Standards-Based Translational Research Informatics Platform , 2009, AMIA.

[28]  J. Franklin,et al.  The elements of statistical learning: data mining, inference and prediction , 2005 .

[29]  Nan Jiang,et al.  Doubly Robust Off-policy Value Evaluation for Reinforcement Learning , 2015, ICML.

[30]  Noémie Elhadad,et al.  Identifying and mitigating biases in EHR laboratory tests , 2014, J. Biomed. Informatics.

[31]  R. Fincher,et al.  Clinical significance of extreme elevation of the erythrocyte sedimentation rate. , 1986, Archives of internal medicine.

[32]  I. Kohane,et al.  Biases in electronic health record data due to processes within the healthcare system: retrospective observational study , 2018, British Medical Journal.

[33]  Rosalind W. Picard,et al.  Deep Reinforcement Learning for Optimal Critical Care Pain Management with Morphine using Dueling Double-Deep Q Networks , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[34]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[35]  Stef van Buuren,et al.  MICE: Multivariate Imputation by Chained Equations in R , 2011 .

[36]  Bekele Afessa,et al.  The influence of missing components of the Acute Physiology Score of APACHE III on the measurement of ICU performance , 2005, Intensive Care Medicine.

[37]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[38]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[39]  Gomathi Krishnan,et al.  Discordant aPTT and Anti-Xa Values and Outcomes in Hospitalized Patients Treated with Intravenous Unfractionated Heparin , 2013, The Annals of pharmacotherapy.

[40]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[41]  Anis Sharafoddini,et al.  A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study , 2018, JMIR medical informatics.

[42]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[43]  Aldo A. Faisal,et al.  The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care , 2018, Nature Medicine.

[44]  Srivatsan Srinivasan,et al.  Evaluating Reinforcement Learning Algorithms in Observational Health Settings , 2018, ArXiv.

[45]  Hoang Minh Le,et al.  Empirical Analysis of Off-Policy Policy Evaluation for Reinforcement Learning , 2019 .