Automating endoscopic camera motion for teleoperated minimally invasive surgery using inverse reinforcement learning

During a laparoscopic surgery, an endoscopic camera is used to provide visual feedback of the surgery to the surgeon and is controlled by a skilled assisting surgeon or a nurse. However, in robot-assisted teleoperated systems such as the daVinci surgical system, the same control lies with the operating surgeons. This results in an added task of constantly changing view point of the endoscope which can be disruptive and also increase the cognitive load on the surgeons. The work presented in this thesis aims to provide an approach that results in an intelligent camera control for such systems using machine learning algorithms. A particular task of pick and place was selected to demonstrate this approach. To add a layer of intelligence to the endoscope, the task was classified into subtasks representing the intent of the user. Neural networks with long short term memory cells (LSTMs) were trained to classify the motion of the instruments in the subtasks and a policy was calculated for each subtask using inverse reinforcement learning (IRL). Since current surgical robots do not enable the movement of the camera and instruments simultaneously, an expert data set was unavailable that could be used to train the models. Hence, a user study was conducted in which the participants were asked to complete the task of picking and placing a ring on a peg in a 3-D immersive simulation environment created using CHAI libraries. A virtual reality headset, Oculus Rift, was used during the study to track the head movements of the users to obtain their view points while they performed the task. This was considered to be expert data and was used to train the algorithm to automate the endoscope motion. A 71.3% accuracy was obtained for the classification of the task into 4 subtasks and the inverse reinforcement learning resulted in an automated trajectory of the endoscope which was 94.7% similar to the human trajectories collected demonstrating that the approach provided in thesis can be used to automate endoscopic motion similar to a skilled assisting surgeon.

[1]  Gregory Auner,et al.  Eye Gaze Tracking for Endoscopic Camera Positioning: An Application of a Hardware/Software Interface Developed to Automate Aesop , 2008, MMVR.

[2]  Andrew W. Senior,et al.  Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition , 2014, ArXiv.

[3]  Philippe Cinquin,et al.  ViKY Robotic Scope Holder: Initial Clinical Experience and Preliminary Results Using Instrument Tracking , 2010, IEEE/ASME Transactions on Mechatronics.

[4]  Pieter Abbeel,et al.  Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Manuel Lopes,et al.  Active Learning for Reward Estimation in Inverse Reinforcement Learning , 2009, ECML/PKDD.

[6]  Monica C. Vroman.  MAXIMUM LIKELIHOOD INVERSE REINFORCEMENT LEARNING , 2014 .

[7]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[8]  Peter Kazanzides,et al.  Software Architecture of the Da Vinci Research Kit , 2017, 2017 First IEEE International Conference on Robotic Computing (IRC).

[9]  Abhilash Pandya,et al.  A Review of Camera Viewpoint Automation in Robotic and Laparoscopic Surgery , 2014, Robotics.

[10]  Michael C. Yip,et al.  Robot Autonomy for Surgery , 2017, The Encyclopedia of Medical Robotics.

[11]  Paula Gomes,et al.  Surgical robotics: Reviewing the past, analysing the present, imagining the future , 2011 .

[12]  A. Lanfranco,et al.  Robotic Surgery: A Current Perspective , 2004, Annals of surgery.

[13]  Pieter Abbeel,et al.  Benchmarking Deep Reinforcement Learning for Continuous Control , 2016, ICML.

[14]  Amnon Shashua,et al.  Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving , 2016, ArXiv.

[15]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Sergey Levine,et al.  Continuous Inverse Optimal Control with Locally Optimal Examples , 2012, ICML.

[18]  Klaus Radermacher,et al.  Development of an automatic surgical holding system based on ergonomic analysis , 1997, CVRMed.

[19]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[20]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[21]  Anind K. Dey,et al.  Maximum Entropy Inverse Reinforcement Learning , 2008, AAAI.

[22]  Stefan Schaal,et al.  Reinforcement Learning for Humanoid Robotics , 2003 .

[23]  Daniel J Scott,et al.  Design of a Proficiency-Based Skills Training Curriculum for the Fundamentals of Laparoscopic Surgery , 2007, Surgical innovation.

[24]  Andreas Bihlmaier,et al.  Learning Dynamic Spatial Relations , 2016, Springer Fachmedien Wiesbaden.

[25]  Gregory D. Hager,et al.  Human-Machine Collaborative surgery using learned models , 2011, 2011 IEEE International Conference on Robotics and Automation.

[26]  G. Buess,et al.  Trocar and instrument positioning system TISKA , 1999, Surgical Endoscopy.

[27]  Dong-Soo Kwon,et al.  Intelligent Laparoscopic Assistant Robot through Surgery Task Model: How to Give Intelligence to Medical Robots , 2008 .

[28]  Darwin G. Caldwell,et al.  Reinforcement Learning in Robotics: Applications and Real-World Challenges , 2013, Robotics.

[29]  Axel Krieger,et al.  Semi-autonomous electrosurgery for tumor resection using a multi-degree of freedom electrosurgical tool and visual servoing , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[30]  Yuan-Fang Wang,et al.  Image analysis for automated tracking in robot-assisted endoscopic surgery , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[31]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[32]  Eyal Amir,et al.  Bayesian Inverse Reinforcement Learning , 2007, IJCAI.

[33]  Gregory D. Hager,et al.  Automatic Detection and Segmentation of Robot-Assisted Surgical Motions , 2005, MICCAI.

[34]  Axel Krieger,et al.  Smart Tissue Anastomosis Robot (STAR): A Vision-Guided Robotics System for Laparoscopic Suturing , 2014, IEEE Transactions on Biomedical Engineering.

[35]  P. Cimiano,et al.  Comparing Hidden Markov Models and Long Short Term Memory Neural Networks for Learning Action Representations , 2016, MOD.

[36]  Gregory W. Auner,et al.  Performance of basic manipulation and intracorporeal suturing tasks in a robotic surgical system: single- versus dual-monitor views , 2009, Surgical Endoscopy.

[37]  Brett Browning,et al.  A survey of robot learning from demonstration , 2009, Robotics Auton. Syst..

[38]  M. Stella Atkins,et al.  Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment , 2004, ETRA.

[39]  G. Hirzinger,et al.  Real-time visual servoing for laparoscopic surgery. Controlling robot motion with color image segmentation , 1997, IEEE Engineering in Medicine and Biology Magazine.

[40]  Prashant Doshi,et al.  A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress , 2018, Artif. Intell..

[41]  Andrew W. Senior,et al.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling , 2014, INTERSPEECH.

[42]  Gregory D. Hager,et al.  A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery , 2017, IEEE Transactions on Biomedical Engineering.

[43]  Andrew Y. Ng,et al.  Pharmacokinetics of a novel formulation of ivermectin after administration to goats , 2000, ICML.

[44]  Shahab Eslamian,et al.  Towards the Implementation of an Autonomous Camera Algorithm on the da Vinci Platform , 2016, MMVR.

[45]  Tomasz Rogula,et al.  Training and Credentialing in Robotic Surgery , 2015 .

[46]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[47]  Nima Najmaei,et al.  Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging – techniques and applications , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[48]  Steven M. LaValle,et al.  Head tracking for the Oculus Rift , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[49]  K. M. Deliparaschos,et al.  Evolution of autonomous and semi‐autonomous robotic surgical systems: a review of the literature , 2011, The international journal of medical robotics + computer assisted surgery : MRCAS.

[50]  Ajay Varma Mudunuri,et al.  Autonomous camera control system for surgical robots , 2010 .

[51]  G. Hirzinger,et al.  Self-guided robotic camera control for laparoscopic surgery compared with human camera control. , 1999, American journal of surgery.

[52]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .

[53]  Stephen P. Brooks,et al.  Markov Decision Processes. , 1995 .

[54]  Andrew Howard,et al.  Design and use paradigms for Gazebo, an open-source multi-robot simulator , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[55]  Gregory Wilding,et al.  Cognitive skills assessment during robot‐assisted surgery: separating the wheat from the chaff , 2015, BJU international.

[56]  Roderic A. Grupen,et al.  Robot Learning with Predictions of Operator Intent , 2004, AAAI Technical Report.

[57]  Ryan S. Decker,et al.  Supervised autonomous robotic soft tissue surgery , 2016, Science Translational Medicine.

[58]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[59]  Brijen Thananjeyan,et al.  Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[60]  Holger Voos,et al.  Controller design for quadrotor UAVs using reinforcement learning , 2010, 2010 IEEE International Conference on Control Applications.

[61]  T. Robinson,et al.  Minimally invasive surgery , 1999, European Surgical Research.

[62]  Philippe Cinquin,et al.  Automatic Detection of Instruments in Laparoscopic Images: A First Step Towards High-level Command of Robotic Endoscopic Holders , 2007, The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006..

[63]  Francois Conti,et al.  The CHAI Libraries , 2003 .

[64]  Gary Boone,et al.  Efficient reinforcement learning: model-based Acrobot control , 1997, Proceedings of International Conference on Robotics and Automation.

[65]  C. Nathan,et al.  The voice-controlled robotic assist scope holder AESOP for the endoscopic approach to the sella. , 2006, Skull base : official journal of North American Skull Base Society ... [et al.].

[66]  Anca D. Dragan,et al.  Cooperative Inverse Reinforcement Learning , 2016, NIPS.