Reinforcement Learning for Closed-Loop Propofol Anesthesia: A Human Volunteer Study

Research has demonstrated the efficacy of closed-loop control of anesthesia using the bispectral index (BIS) of the electroencephalogram as the controlled variable, and the development of model-based, patient-adaptive systems has considerably improved anesthetic control. To further explore the use of model-based control in anesthesia, we investigated the application of reinforcement learning (RL) in the delivery of patient-specific, propofol-induced hypnosis in human volunteers. When compared to published performance metrics, RL control demonstrated accuracy and stability, indicating that further, more rigorous clinical study is warranted.

[1]  F S Servin,et al.  TCI compared with manually controlled infusion of propofol: a multicentre study , 1998, Anaesthesia.

[2]  Luc Barvais,et al.  Titration of Propofol for Anesthetic Induction and Maintenance Guided by the Bispectral Index: Closed-loop versus Manual Control: A Prospective, Randomized, Multicenter Study , 2006, Anesthesiology.

[3]  K. Leslie,et al.  For Personal Use. Only Reproduce with Permission from the Lancet , 2022 .

[4]  John N. Tsitsiklis,et al.  Analysis of temporal-difference learning with function approximation , 1996, NIPS 1996.

[5]  G Rolly,et al.  Closed‐loop controlled administration of propofol using bispectral analysis , 1998, Anaesthesia.

[6]  G N Kenny,et al.  Closed-loop control of propofol anaesthesia. , 1999, British journal of anaesthesia.

[7]  E. Kochs,et al.  Time Delay of Index Calculation: Analysis of Cerebral State, Bispectral, and Narcotrend Indices , 2006, Anesthesiology.

[8]  Steven L Shafer,et al.  Influence of Administration Rate on Propofol Plasma–Effect Site Equilibration , 2007, Anesthesiology.

[9]  S. Petersen-Felix,et al.  Fuzzy logic control of mechanical ventilation during anaesthesia. , 1996, British journal of anaesthesia.

[10]  Steven L. Shafer,et al.  Measuring the predictive performance of computer-controlled infusion pumps , 1992, Journal of Pharmacokinetics and Biopharmaceutics.

[11]  A. Absalom,et al.  Closed-loop Control of Anesthesia Using Bispectral Index: Performance Assessment in Patients Undergoing Major Orthopedic Surgery under Combined General and Regional Anesthesia , 2002, Anesthesiology.

[12]  P. Dayan.  The Convergence of TD(λ) for General λ , 1992, Machine Learning.

[13]  A Carregal,et al.  [Intraoperative control of mean arterial pressure and heart rate with alfentanyl with fuzzy logic]. , 2000, Revista espanola de anestesiologia y reanimacion.

[14]  Peter S. Sebel,et al.  Bispectral Analysis Measures Sedation and Memory Effects of Propofol, Midazolam, Isoflurane, and Alfentanil in Healthy Volunteers , 1997, Anesthesiology.

[15]  Michel M. R. F. Struys,et al.  The Accuracy and Clinical Feasibility of a New Bayesian-Based Closed-Loop Control System for Propofol Administration Using the Bispectral Index as a Controlled Variable , 2008, Anesthesia and analgesia.

[16]  F Cantraine,et al.  Administration of propofol by target-controlled infusion in patients undergoing coronary artery surgery. , 1996, Journal of cardiothoracic and vascular anesthesia.

[17]  Leemon C. Baird,et al.  Residual Algorithms: Reinforcement Learning with Function Approximation , 1995, ICML.

[18]  Jinbao Li,et al.  Anesthesia awareness and the bispectral index. , 2008, The New England journal of medicine.

[19]  Mohammad Bagher Shamsollahi,et al.  Estimating the depth of anesthesia using fuzzy soft computation applied to EEG features , 2008, Intell. Data Anal..

[20]  Karen B. Domino,et al.  The Incidence of Awareness During Anesthesia: A Multicenter United States Study , 2004, Anesthesia and analgesia.

[21]  I. Rampil.  A Primer for EEG Signal Processing in Anesthesia , 1998, Anesthesiology.

[22]  Steven L. Shafer,et al.  Comparison of Some Control Strategies for Three-compartment PK/PD Models , 1994 .

[23]  Anthony G. Doufas,et al.  Induction Speed Is Not a Determinant of Propofol Pharmacodynamics , 2004, Anesthesiology.

[24]  A Hoeft,et al.  Surgical Stimulation Shifts EEG Concentration–Response Relationship of Desflurane , 2001, Anesthesiology.

[25]  A Matsuki,et al.  Use of an EEG‐bispectral closed‐loop delivery system for administering propofol , 2000, Acta anaesthesiologica Scandinavica.

[26]  Vijaykumar Gullapalli,et al.  Learning Control Under Extreme Uncertainty , 1992, NIPS.

[27]  Michel M R F Struys,et al.  Performance Evaluation of Two Published Closed-loop Control Systems Using Bispectral Index Monitoring: A Simulation Study , 2004, Anesthesiology.

[28]  Larry D. Pyeatt,et al.  Fuzzy control for closed-loop, patient-specific hypnosis in intraoperative patients: A simulation study , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[29]  K. Leslie,et al.  Closed loop control of sedation for colonoscopy using the Bispectral Index* , 2002, Anaesthesia.

[30]  D R Stanski,et al.  Plasma Concentrations of Alfentanil Required to Supplement Nitrous Oxide Anesthesia for General Surgery , 1986, Anesthesiology.

[31]  M Wood.  Variability of human drug response. , 1989, Anesthesiology.

[32]  A. Burm,et al.  Performance of Computer-Controlled Infusion of Propofol: An Evaluation of Five Pharmacokinetic Parameter Sets , 1995, Anesthesia and analgesia.

[33]  Larry D. Pyeatt,et al.  Intelligent Control of Closed-Loop Sedation in Simulated ICU Patients , 2004, FLAIRS.

[34]  Fredrik Granath,et al.  Mortality Within 2 Years After Surgery in Relation to Low Intraoperative Bispectral Index Values and Preexisting Malignant Disease , 2009, Anesthesia and analgesia.

[35]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[36]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[37]  T E Stanley,et al.  Midazolam and fentanyl continuous infusion anesthesia for cardiac surgery: a comparison of computer-assisted versus manual infusion systems. , 1993, Journal of cardiothoracic and vascular anesthesia.

[38]  S. Shafer,et al.  The Influence of Method of Administration and Covariates on the Pharmacokinetics of Propofol in Adult Volunteers , 1998, Anesthesiology.

[39]  C. Lennmarken,et al.  Awareness during anaesthesia: a prospective case study , 2000, The Lancet.