Vision-Based Surgical Tool Pose Estimation for the da Vinci® Robotic Surgical System

This paper presents an approach to surgical tool tracking using stereo vision for the da Vinci® Surgical Robotic System. The proposed method is based on robot kinematics, computer vision techniques and Bayesian state estimation. The proposed method employs a silhouette rendering algorithm to create virtual images of the surgical tool by generating the silhouette of the defined tool geometry under the da Vinci® robot endoscopes. The virtual rendering method provides the tool representation in image form, which makes it possible to measure the distance between the rendered tool and real tool from endoscopic stereo image streams. Particle Filter algorithm employing the virtual rendering method is then used for surgical tool tracking. The tracking performance is evaluated on an actual da Vinci® surgical robotic system and a ROS/Gazebo-based simulation of the da Vinci® system.

[1]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Markus Vincze,et al.  Vision for Robotics: a tool for model-based object tracking , 2005, IEEE Robotics & Automation Magazine.

[4]  Wyatt S. Newman,et al.  Three-Dimensional Surgical Needle Localization and Tracking Using Stereo Endoscopic Image Streams , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[6]  Sebastian Thrun,et al.  Particle Filters in Robotics , 2002, UAI.

[7]  Rajalida Lipikorn,et al.  Data Reduction for Finding Silhouette Edges on 3D-Animated Model , 2013 .

[8]  Guang-Zhong Yang,et al.  Real-Time 3D Tracking of Articulated Tools for Robotic Surgery , 2016, MICCAI.

[9]  Henrik I. Christensen,et al.  3D textureless object detection and tracking: An edge-based approach , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Mamoru Mitsuishi,et al.  Robust Visual Tracking of Robotic Forceps Under a Microscope Using Kinematic Data Fusion , 2014, IEEE/ASME Transactions on Mechatronics.

[11]  Abdul Ghafoor,et al.  Image Matching Using Distance Transform , 2003, SCIA.

[12]  Oussama Khatib,et al.  Global Localization of Objects via Touch , 2011, IEEE Transactions on Robotics.

[13]  Henrik I. Christensen,et al.  Robust 3D visual tracking using particle filtering on the special Euclidean group: A combined approach of keypoint and edge features , 2011, 2011 IEEE International Conference on Robotics and Automation.

[14]  Rama Chellappa,et al.  Fast directional chamfer matching , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Peter Kazanzides,et al.  Investigation of Attitude Tracking Using an Integrated Inertial and Magnetic Navigation System for Hand-Held Surgical Instruments , 2012, IEEE/ASME Transactions on Mechatronics.

[16]  Russell H. Taylor,et al.  Vision-Based Proximity Detection in Retinal Surgery , 2012, IEEE Transactions on Biomedical Engineering.

[17]  Luc Van Gool,et al.  Robust tracking-by-detection using a detector confidence particle filter , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[18]  A. Knoll,et al.  Contour-based surgical instrument tracking supported by kinematic prediction , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[19]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[20]  Vincent Lepetit,et al.  Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes , 2011, 2011 International Conference on Computer Vision.

[21]  Luc Soler,et al.  Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing , 2003, IEEE Trans. Robotics Autom..

[22]  Richard M. Murray,et al.  A Mathematical Introduction to Robotic Manipulation , 1994 .

[23]  P. Allen,et al.  Marker-less Articulated Surgical Tool Detection , 2012 .

[24]  Gregory D. Hager,et al.  Articulated object tracking by rendering consistent appearance parts , 2009, 2009 IEEE International Conference on Robotics and Automation.