Transferring Know-How for an Autonomous Camera Robotic Assistant

Robotic platforms are taking their place in the operating room because they provide more stability and accuracy during surgery. Although most of these platforms are teleoperated, a lot of research is currently being carried out to design collaborative platforms. The objective is to reduce the surgeon workload through the automation of secondary or auxiliary tasks, which would benefit both surgeons and patients by facilitating the surgery and reducing the operation time. One of the most important secondary tasks is the endoscopic camera guidance, whose automation would allow the surgeon to be concentrated on handling the surgical instruments. This paper proposes a novel autonomous camera guidance approach for laparoscopic surgery. It is based on learning from demonstration (LfD), which has demonstrated its feasibility to transfer knowledge from humans to robots by means of multiple expert showings. The proposed approach has been validated using an experimental surgical robotic platform to perform peg transferring, a typical task that is used to train human skills in laparoscopic surgery. The results show that camera guidance can be easily trained by a surgeon for a particular task. Later, it can be autonomously reproduced in a similar way to one carried out by a human. Therefore, the results demonstrate that the use of learning from demonstration is a suitable method to perform autonomous camera guidance in collaborative surgical robotic platforms.

[1]  Heinz Wörn,et al.  An intelligent and autonomous endoscopic guidance system for minimally invasive surgery , 2011, 2011 IEEE International Conference on Robotics and Automation.

[2]  Nima Najmaei,et al.  Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging – techniques and applications , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[3]  Maria Cuevas-Rodriguez,et al.  Towards a cognitive camera robotic assistant , 2014, 5th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics.

[4]  Wenjun Xu,et al.  Towards transferring skills to flexible surgical robots with programming by demonstration and reinforcement learning , 2016, 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI).

[5]  Darwin G. Caldwell,et al.  Learning and Reproduction of Gestures by Imitation , 2010, IEEE Robotics & Automation Magazine.

[6]  Junzhou Huang,et al.  Deep learning based multi-label classification for surgical tool presence detection in laparoscopic videos , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[7]  Victor F. Muñoz,et al.  Navigation Method for Teleoperated Single-Port Access Surgery With Soft Tissue Interaction Detection , 2018, IEEE Systems Journal.

[8]  Hedyeh Rafii-Tari,et al.  A cooperative control framework for haptic guidance of bimanual surgical tasks based on Learning From Demonstration , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Hongliang Ren,et al.  Motion Planning Based on Learning From Demonstration for Multiple-Segment Flexible Soft Robots Actuated by Electroactive Polymers , 2016, IEEE Robotics and Automation Letters.

[10]  J. Stolzenburg,et al.  Comparison of the FreeHand® robotic camera holder with human assistants during endoscopic extraperitoneal radical prostatectomy , 2011, BJU international.

[11]  G. Hirzinger,et al.  Self-guided robotic camera control for laparoscopic surgery compared with human camera control. , 1999, American journal of surgery.

[12]  Jaime Molina,et al.  Minimally invasive surgery maneuver recognition based on surgeon model , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Francesca Cordella,et al.  Learning by Demonstration for Planning Activities of Daily Living in Rehabilitation and Assistive Robotics , 2017, IEEE Robotics and Automation Letters.

[14]  Juan Carlos Fraile Marinero,et al.  Smart Cable-Driven Camera Robotic Assistant , 2018, IEEE Transactions on Human-Machine Systems.

[15]  Gregory D. Hager,et al.  Motion generation of robotic surgical tasks: Learning from expert demonstrations , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[16]  Daniel B. Jones,et al.  The SAGES Manual of Quality, Outcomes and Patient Safety , 2011 .

[17]  Peter J. Bentley,et al.  An Intelligent Autopilot System that learns piloting skills from human pilots by imitation , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[18]  Thomas Probst,et al.  Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery , 2018, IEEE Robotics and Automation Letters.

[19]  Jesús Morales,et al.  Pivoting motion control for a laparoscopic assistant robot and human clinical trials , 2005, Adv. Robotics.

[20]  Jason J. Corso,et al.  Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection , 2017, IEEE Transactions on Medical Imaging.

[21]  Pieter Abbeel,et al.  Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations , 2010, 2010 IEEE International Conference on Robotics and Automation.

[22]  Dong-Soo Kwon,et al.  Intelligent interaction between surgeon and laparoscopic assistant robot system , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[23]  Ponani S. Gopalakrishnan,et al.  Clustering via the Bayesian information criterion with applications in speech recognition , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[24]  Andre Schiele,et al.  Using learning from demonstration to generate real-time guidance for haptic shared control , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[25]  Guang-Zhong Yang,et al.  Gaze contingent control for an articulated mechatronic laparoscope , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.