Autonomous Tissue Retraction in Robotic Assisted Minimally Invasive Surgery – A Feasibility Study

In this letter, we describe a novel framework for planning and executing semi-autonomous tissue retraction in minimally invasive robotic surgery. The approach is aimed at removing tissue flaps or connective tissue from the surgical area autonomously, thus exposing the underlying anatomical structures. First, a deep neural network is used to analyse the endoscopic image and detect candidate tissue flaps obstructing the surgical field. A procedural algorithm for planning and executing the retraction gesture is then developed from extended discussions with clinicians. Experimental validation, carried out on a DaVinci Research Kit, shows an average 25% increase of the visible background after retraction. Another significant contribution of this letter is a dataset containing 1,080 labelled surgical stereo images and the associated depth maps, representing tissue flaps in different scenarios. The work described in this letter is a fundamental step towards the autonomous execution of tissue retraction, and the first example of simultaneous use of deep learning and procedural algorithms. The same framework could be applied to a wide range of autonomous tasks, such as debridement and placement of laparoscopic clips.

[1]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[2]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Danail Stoyanov,et al.  Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy , 2019, International Journal of Computer Assisted Radiology and Surgery.

[4]  Elena De Momi,et al.  Automated Pick-Up of Suturing Needles for Robotic Surgical Assistance , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Ron Alterovitz,et al.  Toward automated tissue retraction in robot-assisted surgery , 2010, 2010 IEEE International Conference on Robotics and Automation.

[6]  Richard I. Hartley,et al.  Theory and Practice of Projective Rectification , 1999, International Journal of Computer Vision.

[7]  Klaus H. Maier-Hein,et al.  Brain Tumor Segmentation Using Large Receptive Field Deep Convolutional Neural Networks , 2017, Bildverarbeitung für die Medizin.

[8]  Riccardo Muradore,et al.  Development of a Cognitive Robotic System for Simple Surgical Tasks , 2015 .

[9]  Imre J. Rudas,et al.  Ontology-Based Surgical Subtask Automation, Automating Blunt Dissection , 2018, J. Medical Robotics Res..

[10]  F. Anderhuber,et al.  Flexibility of Thiel’s embalmed cadavers: the explanation is probably in the muscles , 2011, Surgical and Radiologic Anatomy.

[11]  Danail Stoyanov,et al.  EasyLabels: weak labels for scene segmentation in laparoscopic videos , 2019, International Journal of Computer Assisted Radiology and Surgery.

[12]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[14]  Ryan K. Orosco,et al.  SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics , 2020, IEEE Robotics and Automation Letters.

[15]  Axel Krieger,et al.  Development and Feasibility of a Robotic Laparoscopic Clipping Tool for Wound Closure and Anastomosis. , 2018, Journal of medical devices.

[16]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[17]  Imre J. Rudas,et al.  Towards surgical subtask automation — Blunt dissection , 2017, 2017 IEEE 21st International Conference on Intelligent Engineering Systems (INES).

[18]  Holger Roth,et al.  Unsupervised segmentation of 3D medical images based on clustering and deep representation learning , 2018, Medical Imaging.

[19]  May Liu,et al.  A Review of Training Research and Virtual Reality Simulators for the da Vinci Surgical System , 2015, Teaching and learning in medicine.

[20]  Pieter Abbeel,et al.  Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[21]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Pieter Abbeel,et al.  Autonomous multilateral debridement with the Raven surgical robot , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Imre J. Rudas,et al.  Surgical subtask automation — Soft tissue retraction , 2018, 2018 IEEE 16th World Symposium on Applied Machine Intelligence and Informatics (SAMI).

[24]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[25]  J. Anger,et al.  Safety, efficiency and learning curves in robotic surgery: a human factors analysis , 2016, Surgical Endoscopy.

[26]  P R C Steele,et al.  Current and future practices in surgical retraction. , 2013, The surgeon : journal of the Royal Colleges of Surgeons of Edinburgh and Ireland.

[27]  Jim C Hu,et al.  Perioperative complications of laparoscopic and robotic assisted laparoscopic radical prostatectomy. , 2006, The Journal of urology.

[28]  Milan Sonka,et al.  3D Slicer as an image computing platform for the Quantitative Imaging Network. , 2012, Magnetic resonance imaging.

[29]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[30]  Giancarlo Ferrigno,et al.  “Deep-Onto” network for surgical workflow and context recognition , 2018, International Journal of Computer Assisted Radiology and Surgery.

[31]  Henry C. Lin,et al.  JHU-ISI Gesture and Skill Assessment Working Set ( JIGSAWS ) : A Surgical Activity Dataset for Human Motion Modeling , 2014 .

[32]  Kenneth Y. Goldberg,et al.  Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Brian S. Peters,et al.  Review of emerging surgical robotic technology , 2018, Surgical Endoscopy.

[34]  J. Carmichael,et al.  Comparison of open, laparoscopic, and robotic approaches for total abdominal colectomy , 2016, Surgical Endoscopy.

[35]  Manbae Kim,et al.  2D to 3D stereoscopic conversion: depth-map estimation in a 2D single-view image , 2007, SPIE Optical Engineering + Applications.

[36]  Kenneth Y. Goldberg,et al.  An interchangeable surgical instrument system with application to supervised automation of multilateral tumor resection , 2016, 2016 IEEE International Conference on Automation Science and Engineering (CASE).

[37]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.