Three-Dimensional Surgical Needle Localization and Tracking Using Stereo Endoscopic Image Streams

This paper presents algorithms for three-dimensional tracking of surgical needles using the stereo endoscopic camera images obtained from the da Vinci® Surgical Robotic System. The proposed method employs Bayesian state estimation, computer vision techniques, and robot kinematics. A virtual needle rendering procedure is implemented to create simulated images of the surgical needle under the da Vinci ® robot endoscope, which makes it possible to measure the similarity between the rendered needle image and the real needle. A particle filter algorithm using the mentioned techniques is then used for tracking the surgical needle. The performance of the tracking is experimentally evaluated using an actual da Vinci® surgical robotic system and quantitatively validated in a ROS/Gazebo simulation thereof.

[1]  A. Lanfranco,et al.  Robotic Surgery: A Current Perspective , 2004, Annals of surgery.

[2]  Wyatt S. Newman,et al.  Automatic initialization and dynamic tracking of surgical suture threads , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Sebastian Thrun,et al.  Particle Filters in Robotics , 2002, UAI.

[4]  Michel Mourad,et al.  Minimally Invasive Video-assisted Thyroidectomy: Multiinstitutional Experience , 2002, World Journal of Surgery.

[5]  Pieter Abbeel,et al.  Autonomous multilateral debridement with the Raven surgical robot , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[6]  Murat Cenk Cavusoglu,et al.  Needle Grasp and Entry Port Selection for Automatic Execution of Suturing Tasks in Robotic Minimally Invasive Surgery , 2016, IEEE Transactions on Automation Science and Engineering.

[7]  Murat Cenk Cavusoglu,et al.  Needle path planning for autonomous robotic surgical suturing , 2013, 2013 IEEE International Conference on Robotics and Automation.

[8]  Luc Van Gool,et al.  Robust tracking-by-detection using a detector confidence particle filter , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[9]  Santosh Iyer,et al.  A single arm, single camera system for automated suturing , 2013, 2013 IEEE International Conference on Robotics and Automation.

[10]  Benjamin R. Lee,et al.  Comparison of robotic versus laparoscopic skills: is there a difference in the learning curve? , 2002, Urology.

[11]  K. M. Deliparaschos,et al.  Evolution of autonomous and semi‐autonomous robotic surgical systems: a review of the literature , 2011, The international journal of medical robotics + computer assisted surgery : MRCAS.

[12]  Murat Cenk Cavusoglu,et al.  Needle-tissue interaction force state estimation for robotic surgical suturing , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[13]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[14]  G. Kitagawa.  Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models , 1996 .

[15]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[16]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[17]  S Duke Herrell,et al.  Robotic-assisted laparoscopic prostatectomy: do minimally invasive approaches offer significant advantages? , 2005, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[18]  Luc Soler,et al.  Circular needle and needle-holder localization for computer-aided suturing in laparoscopic surgery , 2005, SPIE Medical Imaging.

[19]  Woong Youn Chung,et al.  Robotic thyroid surgery using a gasless, transaxillary approach and the da Vinci S system: the operative outcomes of 338 consecutive patients. , 2009, Surgery.

[20]  Carsten Steger,et al.  An Unbiased Detector of Curvilinear Structures , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Kenneth Y. Goldberg,et al.  Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Wyatt S. Newman,et al.  Real-Time Visual Tracking of Dynamic Surgical Suture Threads , 2018, IEEE Transactions on Automation Science and Engineering.

[23]  Henrik I. Christensen,et al.  Robust 3D visual tracking using particle filtering on the special Euclidean group: A combined approach of keypoint and edge features , 2011, 2011 IEEE International Conference on Robotics and Automation.

[24]  Murat Cenk Cavusoglu,et al.  Modeling of needle-tissue interaction forces during surgical suturing , 2012, 2012 IEEE International Conference on Robotics and Automation.

[25]  Mamoru Mitsuishi,et al.  Preliminary study of needle tracking in a microsurgical robotic system for automated operations , 2013, 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013).

[26]  F. Mohr,et al.  Computer-enhanced "robotic" cardiac surgery: experience in 148 patients. , 2001, The Journal of thoracic and cardiovascular surgery.