Real-Time Visual Tracking of Dynamic Surgical Suture Threads

In order to realize many of the potential benefits associated with robotically assisted minimally invasive surgery, the robot must be more than a remote controlled device. Currently, using a surgical robot can be challenging, fatiguing, and time-consuming. Teaching the robot to actively assist surgical tasks, such as suturing, has the potential to vastly improve both the patient’s outlook and the surgeon’s efficiency. One obstacle to completing surgical sutures autonomously is the difficulty in tracking surgical suture threads. This paper presents novel stereo image processing algorithms for the detection, initialization, and tracking of a surgical suture thread. A nonuniform rational B-spline (NURBS) curve is used to model a thin, deformable, and dynamic length thread. The NURBS model is initialized and grown from a single selected point located on the thread. The NURBS curve is optimized by minimizing the image matching energy between the projected stereo NURBS image and the segmented thread image. The algorithms are evaluated using suture threads, a calibrated test pattern, and a simulated thread image. In addition, the accuracies of the algorithms presented are validated as they track a suture thread undergoing translation, deformation, and apparent length changes. All of the tracking is in real time. Note to Practitioners—The problem of tracking a surgical suture thread was addressed in this paper. Since the suture thread is highly deformable, any tracking algorithm must be robust to intersections, occlusions, knot tying, and length changes. The detection algorithm introduced in this paper is capable of distinguishing different threads when they intersect. The tracking algorithm presented here demonstrates that it is possible, using polynomial curves, to track a suture thread as it deforms, becomes occluded, changes length, and even ties a knot in real time. The detection algorithm can enhance directional thin features, while the polynomial curve modeling can track any string-like structure. Further integration of the polynomial curve with a feed-forward thread model could improve the stability and robustness of the thread tracking.

[1]  Adrien Bartoli,et al.  Reconstruction of 3D Curves for Quality Control , 2007, SCIA.

[2]  Blake Hannaford,et al.  Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model , 2006, IEEE Transactions on Biomedical Engineering.

[3]  J. M. Sackier,et al.  Robotically assisted laparoscopic surgery , 2008, Surgical Endoscopy.

[4]  Tien Yin Wong,et al.  Red lesion detection in retinal fundus images using Frangi-based filters , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[5]  Carsten Steger,et al.  An Unbiased Detector of Curvilinear Structures , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Ravinda G. N. Meegama,et al.  NURBS snakes , 2003, Image Vis. Comput..

[8]  Aamir Saeed Malik,et al.  Subcutaneous veins detection and backprojection method using Frangi vesselness filter , 2015, 2015 IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE).

[9]  Colin D. Bicknell,et al.  Robust Catheter and Guidewire Tracking Using B-Spline Tube Model and Pixel-Wise Posteriors , 2016, IEEE Robotics and Automation Letters.

[10]  Wyatt S. Newman,et al.  Automatic initialization and dynamic tracking of surgical suture threads , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Rachid Deriche,et al.  A Review of Statistical Approaches to Level Set Segmentation: Integrating Color, Texture, Motion and Shape , 2007, International Journal of Computer Vision.

[12]  Darius Burschka,et al.  Human-machine skill transfer extended by a scaffolding framework , 2008, 2008 IEEE International Conference on Robotics and Automation.

[13]  Nikos Komodakis,et al.  Fast, Approximately Optimal Solutions for Single and Dynamic MRFs , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  J. Sethian,et al.  A Fast Level Set Method for Propagating Interfaces , 1995 .

[15]  Anthony J. Yezzi,et al.  Detecting Curves with Unknown Endpoints and Arbitrary Topology Using Minimal Paths , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Alois Knoll,et al.  Framework of automatic robot surgery system using Visual servoing , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[18]  Melvyn L. Smith,et al.  High speed, multi-scale tracing of curvilinear features with automated scale selection and enhanced orientation computation , 2010, 2010 International Conference on High Performance Computing & Simulation.

[19]  Edgar J. Lobaton,et al.  Continuous shape estimation of continuum robots using X-ray images , 2013, 2013 IEEE International Conference on Robotics and Automation.

[20]  Alois Knoll,et al.  Automation of tissue piercing using circular needles and vision guidance for computer aided laparoscopic surgery , 2010, 2010 IEEE International Conference on Robotics and Automation.

[21]  Nassir Navab,et al.  Discrete tracking of parametrized curves , 2009, CVPR 2009.

[22]  Pieter Abbeel,et al.  Tracking deformable objects with point clouds , 2013, 2013 IEEE International Conference on Robotics and Automation.

[23]  Mitra Basu,et al.  Gaussian-based edge-detection methods - a survey , 2002, IEEE Trans. Syst. Man Cybern. Part C.

[24]  Mamoru Mitsuishi,et al.  Trajectory planning under different initial conditions for surgical task automation by learning from demonstration , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[25]  M. Bonfe,et al.  Towards automated surgical robotics: A requirements engineering approach , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[26]  Les A. Piegl,et al.  The NURBS book (2nd ed.) , 1997 .

[27]  Haibin Ling,et al.  Curvilinear Structure Tracking by Low Rank Tensor Approximation with Model Propagation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Kenneth Y. Goldberg,et al.  Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Laurent D. Cohen,et al.  Tubular Structure Segmentation Based on Minimal Path Method and Anisotropic Enhancement , 2011, International Journal of Computer Vision.

[30]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[31]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[32]  Mamoru Mitsuishi,et al.  Preliminary study of needle tracking in a microsurgical robotic system for automated operations , 2013, 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013).

[33]  Tat-Jen Cham,et al.  Stereo coupled active contours , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Gregory D. Hager,et al.  3D thread tracking for robotic assistance in tele-surgery , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[35]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[36]  Santosh Iyer,et al.  A single arm, single camera system for automated suturing , 2013, 2013 IEEE International Conference on Robotics and Automation.

[37]  Alan C. Bovik,et al.  A Steerable, Multiscale Singularity Index , 2013, IEEE Signal Processing Letters.

[38]  Gregory D. Hager,et al.  Deformable Tracking of Textured Curvilinear Objects , 2012, BMVC.

[39]  James F. O'Brien,et al.  Modeling and perception of deformable one-dimensional objects , 2011, 2011 IEEE International Conference on Robotics and Automation.

[40]  Wyatt S. Newman,et al.  A novel vision guided knot-tying method for autonomous robotic surgery , 2014, 2014 IEEE International Conference on Automation Science and Engineering (CASE).

[41]  Shuji Hashimoto,et al.  Fast crack detection method for large-size concrete surface images using percolation-based image processing , 2010, Machine Vision and Applications.

[42]  Axel Krieger,et al.  Smart Tissue Anastomosis Robot (STAR): A Vision-Guided Robotics System for Laparoscopic Suturing , 2014, IEEE Transactions on Biomedical Engineering.

[43]  Isabelle Bloch,et al.  A review of 3D vessel lumen segmentation techniques: Models, features and extraction schemes , 2009, Medical Image Anal..

[44]  Mohamed Cheriet,et al.  Vesselwalker: Coronary arteries segmentation using random walks and hessian-based vesselness filter , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[45]  A. Knoll,et al.  Human-computer interfaces for interaction with surgical tools in robotic surgery , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).