Concurrent Segmentation and Localization for Tracking of Surgical Instruments

Real-time instrument tracking is a crucial requirement for various computer-assisted interventions. To overcome problems such as specular reflection and motion blur, we propose a novel method that takes advantage of the interdependency between localization and segmentation of the surgical tool. In particular, we reformulate the 2D pose estimation as a heatmap regression and thereby enable a robust, concurrent regression of both tasks via deep learning. Throughout experimental results, we demonstrate that this modeling leads to a significantly better performance than directly regressing the tool position and that our method outperforms the state-of-the-art on a Retinal Microsurgery benchmark and the MICCAI EndoVis Challenge 2015.

[1]  Jason J. Corso,et al.  Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection , 2017, IEEE Transactions on Medical Imaging.

[2]  Nassir Navab,et al.  Sensor substitution for video-based action recognition , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[3]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[4]  Sébastien Ourselin,et al.  Real-Time Segmentation of Non-rigid Surgical Tools Based on Deep Learning and Tracking , 2016, CARE@MICCAI.

[5]  Russell H. Taylor,et al.  Unified Detection and Tracking of Instruments during Retinal Microsurgery , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Pascal Fua,et al.  Fast Part-Based Classification for Instrument Detection in Minimally Invasive Surgery , 2014, MICCAI.

[7]  Andrew Zisserman,et al.  Flowing ConvNets for Human Pose Estimation in Videos , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[8]  P. Allen,et al.  Marker-less Articulated Surgical Tool Detection , 2012 .

[9]  Junzhou Huang,et al.  Instrument Tracking via Online Learning in Retinal Microsurgery , 2014, MICCAI.

[10]  Guang-Zhong Yang,et al.  Real-Time 3D Tracking of Articulated Tools for Robotic Surgery , 2016, MICCAI.

[11]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[12]  Shahram Payandeh,et al.  Visual Tracking of Laparoscopic Instruments , 2014 .

[13]  Stefanie Speidel,et al.  Automatic classification of minimally invasive instruments based on endoscopic image sequences , 2009, Medical Imaging.

[14]  Nassir Navab,et al.  Deep Residual Learning for Instrument Segmentation in Robotic Surgery , 2017, MLMI@MICCAI.

[15]  Nassir Navab,et al.  Real-time localization of articulated surgical instruments in retinal microsurgery , 2016, Medical Image Anal..

[16]  Nassir Navab,et al.  Detection of articulated instruments in retinal microsurgery , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[17]  Xiaowei Zhou,et al.  Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Bernt Schiele,et al.  Detecting Surgical Tools by Modelling Local Appearance and Global Shape , 2015, IEEE Transactions on Medical Imaging.

[19]  Nassir Navab,et al.  Surgical Tool Tracking and Pose Estimation in Retinal Microsurgery , 2015, MICCAI.

[20]  Nassir Navab,et al.  Real-Time Online Adaption for Robust Instrument Tracking and Pose Estimation , 2016, MICCAI.

[21]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[23]  Nassir Navab,et al.  Deeper Depth Prediction with Fully Convolutional Residual Networks , 2016, 2016 Fourth International Conference on 3D Vision (3DV).