Augmented Reality Predictive Displays to Help Mitigate the Effects of Delayed Telesurgery

Surgical robots offer the exciting potential for remote telesurgery, but advances are needed to make this technology efficient and accurate to ensure patient safety. Achieving these goals is hindered by the deleterious effects of latency between the remote operator and the bedside robot. Predictive displays have found success in overcoming these effects by giving the operator immediate visual feedback. However, previously developed predictive displays can not be directly applied to telesurgery due to the unique challenges in tracking the 3D geometry of the surgical environment. In this paper, we present the first predictive display for teleoperated surgical robots. The predicted display is stereoscopic, utilizes Augmented Reality (AR) to show the predicted motions alongside the complex tissue found in-situ within surgical environments, and overcomes the challenges in accurately tracking slave-tools in real-time. We call this a Stereoscopic AR Predictive Display (SARPD). To test the SARPD’s performance, we conducted a user study with ten participants on the da Vinci® Surgical System. The results showed with statistical significance that using SARPD decreased time to complete task while having no effect on error rates when operating under delay.

[1]  Russell H. Taylor,et al.  Simple Biomanipulation Tasks with 'Steady Hand' Cooperative Manipulator , 2003, MICCAI.

[2]  Michael C. Yip,et al.  Robot Autonomy for Surgery , 2017, The Encyclopedia of Medical Robotics.

[3]  Thomas B. Sheridan,et al.  Supervisory control of remote manipulation , 1967, IEEE Spectrum.

[4]  Mahdi Tavakoli,et al.  Performance Analysis of a Haptic Telemanipulation Task under Time Delay , 2011, Adv. Robotics.

[5]  W R Ferrell.  Delayed Force Feedback1 , 1966, Human factors.

[6]  Mahdi Tavakoli,et al.  Performance analysis of a manipulation task in time-delayed teleoperation , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[7]  Jean-Jacques E. Slotine,et al.  Using wave variables for system analysis and robot control , 1997, Proceedings of International Conference on Robotics and Automation.

[8]  Won S. Kim,et al.  The phantom robot: predictive displays for teleoperation with time delay , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[9]  Tsuneo Yoshikawa,et al.  Ground-space bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition , 2004, IEEE Transactions on Robotics and Automation.

[10]  Austin Reiter,et al.  Feature Classification for Tracking Articulated Surgical Tools , 2012, MICCAI.

[11]  M. Anvari,et al.  Establishment of the World's First Telerobotic Remote Surgical Service: For Provision of Advanced Laparoscopic Surgery in a Rural Community , 2005, Annals of surgery.

[12]  Mark W. Spong,et al.  Bilateral control of teleoperators with time delay , 1988, Proceedings of the 1988 IEEE International Conference on Systems, Man, and Cybernetics.

[13]  C Y Nguan,et al.  Robotic pyeloplasty using internet protocol and satellite network‐based telesurgery , 2008, The international journal of medical robotics + computer assisted surgery : MRCAS.

[14]  A. Lanfranco,et al.  Robotic Surgery: A Current Perspective , 2004, Annals of surgery.

[15]  John F. Canny,et al.  Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Jacques Felblinger,et al.  Determination of the latency effects on surgical performance and the acceptable latency levels in telesurgery using the dV-Trainer® simulator , 2014, Surgical Endoscopy.

[17]  Lydia E. Kavraki,et al.  Treatment planning for a radiosurgical system with general kinematics , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[18]  Ran Hao,et al.  Vision-Based Surgical Tool Pose Estimation for the da Vinci® Robotic Surgical System , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Masaru Uchiyama,et al.  Model-based space robot teleoperation of ETS-VII manipulator , 2004, IEEE Transactions on Robotics and Automation.

[20]  Allison M. Okamura,et al.  Time-delayed teleoperation for interaction with moving objects in space , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[21]  Jacques Marescaux,et al.  Transatlantic robot-assisted telesurgery , 2001, Nature.

[22]  Guang-Zhong Yang,et al.  Real-Time 3D Tracking of Articulated Tools for Robotic Surgery , 2016, MICCAI.

[23]  Blake Hannaford,et al.  Teleoperation of a Surgical Robot Via Airborne Wireless Radio and Transatlantic Internet Links , 2007, FSR.

[24]  Mark W. Spong,et al.  Bilateral teleoperation: An historical survey , 2006, Autom..

[25]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Peter Kazanzides,et al.  ARssist: augmented reality on a head-mounted display for the first assistant in robotic surgery , 2018, Healthcare technology letters.