Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgery

Deep Reinforcement Learning (DRL) is a viable solution for automating repetitive surgical subtasks due to its ability to learn complex behaviours in a dynamic environment. This task automation could lead to reduced surgeon’s cognitive workload, increased precision in critical aspects of the surgery, and fewer patient-related complications. However, current DRL methods do not guarantee any safety criteria as they maximise cumulative rewards without considering the risks associated with the actions performed. Due to this limitation, the application of DRL in the safety-critical paradigm of robot-assisted Minimally Invasive Surgery (MIS) has been constrained. In this work, we introduce a Safe-DRL framework that incorporates safety constraints for the automation of surgical subtasks via DRL training. We validate our approach in a virtual scene that replicates a tissue retraction task commonly occurring in multiple phases of an MIS. Furthermore, to evaluate the safe behaviour of the robotic arms, we formulate a formal verification tool for DRL methods that provides the probability of unsafe configurations. Our results indicate that a formal analysis guarantees safety with high confidence such that the robotic instruments operate within the safe workspace and avoid hazardous interaction with other anatomical structures.

[1]  Javier García,et al.  A comprehensive survey on safe reinforcement learning , 2015, J. Mach. Learn. Res..

[2]  Imre J. Rudas,et al.  Surgical subtask automation — Soft tissue retraction , 2018, 2018 IEEE 16th World Symposium on Applied Machine Intelligence and Informatics (SAMI).

[3]  Herke van Hoof,et al.  Addressing Function Approximation Error in Actor-Critic Methods , 2018, ICML.

[4]  Ankur Mehta,et al.  Toward Synergic Learning for Autonomous Manipulation of Deformable Tissues via Surgical Robots: An Approximate Q-Learning Approach , 2019, 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob).

[5]  P R C Steele,et al.  Current and future practices in surgical retraction. , 2013, The surgeon : journal of the Royal Colleges of Surgeons of Edinburgh and Ireland.

[6]  Ron Alterovitz,et al.  Toward automated tissue retraction in robot-assisted surgery , 2010, 2010 IEEE International Conference on Robotics and Automation.

[7]  Brijen Thananjeyan,et al.  Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[8]  P. Fiorini,et al.  Soft Tissue Simulation Environment to Learn Manipulation Tasks in Autonomous Robotic Surgery* , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Inderjit S. Dhillon,et al.  Towards Fast Computation of Certified Robustness for ReLU Networks , 2018, ICML.

[10]  Dennis Kundrat,et al.  Collaborative Robot-Assisted Endovascular Catheterization with Generative Adversarial Imitation Learning , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Junfeng Yang,et al.  Efficient Formal Safety Analysis of Neural Networks , 2018, NeurIPS.

[12]  Evan Dekker,et al.  Empirical evaluation methods for multiobjective reinforcement learning algorithms , 2011, Machine Learning.

[13]  John C. Duchi,et al.  Certifying Some Distributional Robustness with Principled Adversarial Training , 2017, ICLR.

[14]  Runzhe Yang,et al.  A Generalized Algorithm for Multi-Objective RL and Policy Adaptation , 2019 .

[15]  Ramon E. Moore.  Interval arithmetic and automatic error analysis in digital computing , 1963 .

[16]  Jacob Rosen,et al.  Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[17]  Ryan K. Orosco,et al.  SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics , 2020, IEEE Robotics and Automation Letters.

[18]  Alessandro Farinelli,et al.  Discrete Deep Reinforcement Learning for Mapless Navigation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Paolo Fiorini,et al.  Formal Verification for Safe Deep Reinforcement Learning in Trajectory Generation , 2020, 2020 Fourth IEEE International Conference on Robotic Computing (IRC).

[20]  Mykel J. Kochenderfer,et al.  Algorithms for Verifying Deep Neural Networks , 2019, Found. Trends Optim..

[21]  Pieter Abbeel,et al.  Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Florian Richter,et al.  Open-Sourced Reinforcement Learning Environments for Surgical Robotics , 2019, ArXiv.

[23]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[24]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[25]  Saeid Nahavandi,et al.  Manipulating Soft Tissues by Deep Reinforcement Learning for Autonomous Robotic Surgery , 2019, 2019 IEEE International Systems Conference (SysCon).

[26]  Alessandro Farinelli,et al.  Formal verification of neural networks for safety-critical tasks in deep reinforcement learning , 2021, UAI.

[27]  Dario Amodei,et al.  Benchmarking Safe Exploration in Deep Reinforcement Learning , 2019 .

[28]  Alejandro F. Frangi,et al.  Autonomous Tissue Retraction in Robotic Assisted Minimally Invasive Surgery – A Feasibility Study , 2020, IEEE Robotics and Automation Letters.

[29]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[30]  Sergey Levine,et al.  Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor , 2018, ICML.