Weakly supervised segmentation for real-time surgical tool tracking

Surgical tool tracking has a variety of applications in different surgical scenarios. Electromagnetic (EM) tracking can be utilised for tool tracking, but the accuracy is often limited by magnetic interference. Vision-based methods have also been suggested; however, tracking robustness is limited by specular reflection, occlusions, and blurriness observed in the endoscopic image. Recently, deep learning-based methods have shown competitive performance on segmentation and tracking of surgical tools. The main bottleneck of these methods lies in acquiring a sufficient amount of pixel-wise, annotated training data, which demands substantial labour costs. To tackle this issue, the authors propose a weakly supervised method for surgical tool segmentation and tracking based on hybrid sensor systems. They first generate semantic labellings using EM tracking and laparoscopic image processing concurrently. They then train a light-weight deep segmentation network to obtain a binary segmentation mask that enables tool tracking. To the authors’ knowledge, the proposed method is the first to integrate EM tracking and laparoscopic image processing for generation of training labels. They demonstrate that their framework achieves accurate, automatic tool segmentation (i.e. without any manual labelling of the surgical tool to be tracked) and robust tool tracking in laparoscopic image sequences.

[1]  Stefanie Speidel,et al.  Automatic classification of minimally invasive instruments based on endoscopic image sequences , 2009, Medical Imaging.

[2]  Sébastien Ourselin,et al.  ToolNet: Holistically-nested real-time segmentation of robotic surgical tools , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[3]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Nicolas Padoy,et al.  Self-Supervised Surgical Tool Segmentation using Kinematic Information , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[5]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[6]  Alexander Rakhlin,et al.  Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning , 2018, bioRxiv.

[7]  George Zaki,et al.  Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution , 2016, Journal of medical imaging.

[8]  Didier Mutter,et al.  Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos , 2018, CVII-STENT/LABELS@MICCAI.

[9]  Shuvra S. Bhattacharyya,et al.  Segmentation of surgical instruments in laparoscopic videos: training dataset generation and deep-learning-based framework , 2019, Medical Imaging.

[10]  Guannan Gao,et al.  Probabilistic Hough Transform , 2011 .

[11]  Jonathan Krause,et al.  Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks , 2018, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[12]  Lena Maier-Hein,et al.  2017 Robotic Instrument Segmentation Challenge , 2019, ArXiv.

[13]  Eugenio Culurciello,et al.  LinkNet: Exploiting encoder representations for efficient semantic segmentation , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[14]  Abhishek Dutta,et al.  The VIA Annotation Software for Images, Audio and Video , 2019, ACM Multimedia.

[15]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[16]  Keith J Murphy,et al.  A novel 3-dimensional electromagnetic guidance system increases intraoperative microwave antenna placement accuracy. , 2017, HPB : the official journal of the International Hepato Pancreato Biliary Association.

[17]  Sébastien Ourselin,et al.  2D-3D Pose Tracking of Rigid Instruments in Minimally Invasive Surgery , 2014, IPCAI.

[18]  Andru Putra Twinanda,et al.  EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos , 2016, IEEE Transactions on Medical Imaging.