CONTROLLING BLOOD GLUCOSE LEVELS IN PATIENTS WITH TYPE 1 DIABETES USING FITTED Q-ITERATIONS AND FUNCTIONAL FEATURES

Type 1 Diabetes is characterized by the lack of insulin-producing beta cells in the pancreas. The artificial pancreas promises to alleviate the burdens of self-management. While the physical components of the system – the continuous glucose monitor and insulin pump – have experienced rapid advances, a technological bottleneck remains in the control algorithm, which is responsible for translating data from the former into instructions for the latter. In this work, we propose to bring machine learning techniques to bear upon the challenges of blood glucose control. Specifically, we employ reinforcement learning to learn an optimal insulin policy. Learning is generalized using nonparametric regression with functional features, exploiting information contained in the shape of the glucose curve. Our algorithm is model-free, data-driven and personalized. In-silico simulations with T1D models demonstrate the potential of the proposed algorithm.

[1]  Ahmad Haidar,et al.  The Artificial Pancreas: How Closed-Loop Control Is Revolutionizing Diabetes , 2016, IEEE Control Systems.

[2]  Roman Hovorka,et al.  Simulation Environment to Evaluate Closed-Loop Insulin Delivery Systems in Type 1 Diabetes , 2010, Journal of diabetes science and technology.

[3]  Ali Cinar,et al.  Artificial Pancreas Systems: An Introduction to the Special Issue , 2018 .

[4]  R. Hovorka,et al.  Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes. , 2004, Physiological measurement.

[5]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[6]  Daniel Nikovski,et al.  Deep reinforcement learning for partial differential equation control , 2017, 2017 American Control Conference (ACC).

[7]  Ali Karimpour,et al.  Agent-based Simulation for Blood Glucose , 2009 .

[8]  Daniel Nikovski,et al.  Learning to control partial differential equations: Regularized Fitted Q-Iteration approach , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[9]  Y. Z. Ider,et al.  Quantitative estimation of insulin sensitivity. , 1979, The American journal of physiology.

[10]  Stavroula G. Mougiakakou,et al.  Personalized tuning of a reinforcement learning control algorithm for glucose regulation , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[11]  Pierre Geurts,et al.  Tree-Based Batch Mode Reinforcement Learning , 2005, J. Mach. Learn. Res..

[12]  Wojciech Zaremba,et al.  OpenAI Gym , 2016, ArXiv.

[13]  Shie Mannor,et al.  Regularized Fitted Q-Iteration for planning in continuous-space Markovian decision problems , 2009, 2009 American Control Conference.

[14]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[15]  Bart De Schutter,et al.  Approximate reinforcement learning: An overview , 2011, 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).

[16]  Meysam Bastani,et al.  Model-Free Intelligent Diabetes Management Using Machine Learning , 2014 .

[17]  Darrell M. Wilson,et al.  A Closed-Loop Artificial Pancreas Using Model Predictive Control and a Sliding Meal Size Estimator , 2009, Journal of diabetes science and technology.

[18]  Mariano De Paula,et al.  Controlling blood glucose variability under uncertainty using reinforcement learning and Gaussian processes , 2015, Appl. Soft Comput..

[19]  Csaba Szepesvári,et al.  Fitted Q-iteration in continuous action-space MDPs , 2007, NIPS.

[20]  Shie Mannor,et al.  Regularized Fitted Q-iteration: Application to Planning , 2008, EWRL.

[21]  Antonio Criminisi,et al.  Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning , 2012, Found. Trends Comput. Graph. Vis..