A Multirobots Teleoperated Platform for Artificial Intelligence Training Data Collection in Minimally Invasive Surgery

Dexterity and perception capabilities of surgical robots may soon be improved by cognitive functions that can support surgeons in decision making and performance monitoring, and enhance the impact of automation within the operating rooms. Nowadays, the basic elements of autonomy in robotic surgery are still not well understood and their mutual interaction is unexplored. Current classification of autonomy encompasses six basic levels: Level 0: no autonomy; Level 1: robot assistance; Level 2: task autonomy; Level 3: conditional autonomy; Level 4: high autonomy. Level 5: full autonomy. The practical meaning of each level and the necessary technologies to move from one level to the next are the subject of intense debate and development. In this paper, we discuss the first outcomes of the European funded project Smart Autonomous Robotic Assistant Surgeon (SARAS). SARAS will develop a cognitive architecture able to make decisions based on pre-operative knowledge and on scene understanding via advanced machine learning algorithms. To reach this ambitious goal that allows us to reach Level 1 and 2, it is of paramount importance to collect reliable data to train the algorithms. We will present the experimental setup to collect the data for a complex surgical procedure (Robotic Assisted Radical Prostatectomy) on very sophisticated manikins (i.e. phantoms of the inflated human abdomen). The SARAS platform allows the main surgeon and the assistant to teleoperate two independent two-arm robots. The data acquired with this platform (videos, kinematics, audio) will be used in our project and will be released (with annotations) for research purposes.

[1]  John Kenneth Salisbury,et al.  The Intuitive/sup TM/ telesurgery system: overview and application , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[2]  S. J. Harris,et al.  The first clinical application of a "hands-on" robotic knee surgery system. , 2001, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[3]  Russell H. Taylor,et al.  Medical robotics in computer-integrated surgery , 2003, IEEE Trans. Robotics Autom..

[4]  Garth H Ballantyne,et al.  The da Vinci telerobotic surgical system: the virtual operative field and telepresence surgery. , 2003, The Surgical clinics of North America.

[5]  Scott Van Appledorn,et al.  Robotic laparoscopic radical prostatectomy: setup and procedural techniques after 150 cases. , 2006, Urology.

[6]  Jürgen Schmidhuber,et al.  A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks , 2006 .

[7]  J. Doyle,et al.  A universal global rating scale for the evaluation of technical skills in the operating room. , 2007, American journal of surgery.

[8]  Jing Ren,et al.  Dynamic 3-D Virtual Fixtures for Minimally Invasive Beating Heart Procedures , 2008, IEEE Transactions on Medical Imaging.

[9]  Russell H. Taylor,et al.  Medical robotics in computer-integrated surgery , 2003, IEEE Trans. Robotics Autom..

[10]  S. Lipsitz,et al.  Comparative effectiveness of minimally invasive vs open radical prostatectomy. , 2009, JAMA.

[11]  Pablo Garcia,et al.  Usability of robotic platforms for remote surgical teleproctoring. , 2009, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[12]  Russell H. Taylor,et al.  New steady-hand Eye Robot with micro-force sensing for vitreoretinal surgery , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[13]  M. Scorsetti,et al.  Applying failure mode effects and criticality analysis in radiotherapy: lessons learned and perspectives of enhancement. , 2010, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[14]  Stefano Stramigioli,et al.  Bilateral Telemanipulation With Time Delays: A Two-Layer Approach Combining Passivity and Transparency , 2011, IEEE Transactions on Robotics.

[15]  A. Marttos,et al.  Surgical telepresence: the usability of a robotic communication platform , 2012, World Journal of Emergency Surgery.

[16]  Jeffrey A. Cadeddu,et al.  Toward Long-Term and Accurate Augmented-Reality for Monocular Endoscopic Videos , 2014, IEEE Transactions on Biomedical Engineering.

[17]  Henry C. Lin,et al.  JHU-ISI Gesture and Skill Assessment Working Set ( JIGSAWS ) : A Surgical Activity Dataset for Human Motion Modeling , 2014 .

[18]  A. Cestari,et al.  Side docking of the da Vinci robotic system for radical prostatectomy: advantages over traditional docking , 2015, Journal of Robotic Surgery.

[19]  Riccardo Muradore,et al.  Development of a Cognitive Robotic System for Simple Surgical Tasks , 2015 .

[20]  Riccardo Muradore,et al.  An Energy Tank-Based Interactive Control Architecture for Autonomous and Teleoperated Robotic Surgery , 2015, IEEE Transactions on Robotics.

[21]  Riccardo Muradore,et al.  A Cognitive Robot Control Architecture for Autonomous Execution of Surgical Tasks , 2016, J. Medical Robotics Res..

[22]  Germain Forestier,et al.  Unsupervised Trajectory Segmentation for Surgical Gesture Recognition in Robotic Training , 2016, IEEE Transactions on Biomedical Engineering.

[23]  Gregory D. Hager,et al.  Temporal Convolutional Networks for Action Segmentation and Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Peter Kazanzides,et al.  Medical robotics—Regulatory, ethical, and legal considerations for increasing levels of autonomy , 2017, Science Robotics.

[25]  Ratna Babu Chinnam,et al.  Soft Boundary Approach for Unsupervised Gesture Segmentation in Robotic-Assisted Surgery , 2017, IEEE Robotics and Automation Letters.

[26]  Chenliang Xu,et al.  TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation , 2017, ArXiv.

[27]  C. Abbou,et al.  LAPAROSCOPIC RADICAL PROSTATECTOMY WITH A REMOTE CONTROLLED ROBOT , 2001, The Journal of urology.

[28]  Riccardo Muradore,et al.  An energy-shared two-layer approach for multi-master-multi-slave bilateral teleoperation systems , 2019, 2019 International Conference on Robotics and Automation (ICRA).