Deep Reinforcement Learning for Optimal Critical Care Pain Management with Morphine using Dueling Double-Deep Q Networks

Opioids are the preferred medications for the treatment of pain in the intensive care unit. While under-treatment leads to unrelieved pain and poor clinical outcomes, excessive use of opioids puts patients at risk of experiencing multiple adverse effects. In this work, we present a sequential decision making framework for opioid dosing based on deep reinforcement learning. It provides real-time clinically interpretable dosing recommendations, personalized according to each patient’s evolving pain and physiological condition. We focus on morphine, one of the most commonly prescribed opioids. To train and evaluate the model, we used retrospective data from the publicly available MIMIC-3 database. Our results demonstrate that reinforcement learning may be used to aid decision making in the intensive care setting by providing personalized pain management interventions.

[1]  David Borsook,et al.  Multi-task multiple kernel machines for personalized pain recognition from functional near-infrared spectroscopy brain signals , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[2]  Peter Szolovits,et al.  MIMIC-III, a freely accessible critical care database , 2016, Scientific Data.

[3]  M. Ashburn,et al.  Cardiac Effects of Opioid Therapy. , 2015, Pain medicine.

[4]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[5]  Ognjen Rudovic,et al.  Personalized Automatic Estimation of Self-Reported Pain Intensity from Facial Expressions , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Ognjen Rudovic,et al.  Physiological and behavioral profiling for nociceptive pain estimation using personalized multitask learning , 2017, ArXiv.

[7]  Peter Szolovits,et al.  Deep Reinforcement Learning for Sepsis Treatment , 2017, ArXiv.

[8]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[9]  M. Lai,et al.  Continuous infusion versus intermittent bolus dosing of morphine: a comparison of analgesia, tolerance, and subsequent voluntary morphine intake. , 2014, Journal of psychiatric research.

[10]  K. Pattinson,et al.  Opioids and the control of respiration. , 2008, British journal of anaesthesia.

[11]  R. Fillingim,et al.  Morphine responses and experimental pain: sex differences in side effects and cardiovascular responses but not analgesia. , 2005, The journal of pain : official journal of the American Pain Society.

[12]  R. Jaeschke,et al.  Clinical Practice Guidelines for the Management of Pain, Agitation, and Delirium in Adult Patients in the Intensive Care Unit , 2013, Critical care medicine.

[13]  Shamim Nemati,et al.  Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  M. Narayanan,et al.  Analgesia in intensive care: part 1 , 2016 .

[16]  Peter Szolovits,et al.  Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach , 2017, MLHC.

[17]  Demis Hassabis,et al.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play , 2018, Science.

[18]  H. Dudley,et al.  Morphine: controlled trial of different methods of administration for postoperative pain relief. , 1980, British medical journal.

[19]  Pratik Shah,et al.  Reinforcement Learning with Action-Derived Rewards for Chemotherapy and Clinical Trial Dosing Regimen Selection , 2018, MLHC.

[20]  Rosalind W. Picard,et al.  Continuous Pain Intensity Estimation from Autonomic Signals with Recurrent Neural Networks , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[21]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.