Development of a Deep Learning-Based Algorithm to Detect the Distal End of a Surgical Instrument

This work aims to develop an algorithm to detect the distal end of a surgical instrument using object detection with deep learning. We employed nine video recordings of carotid endarterectomies for training and testing. We obtained regions of interest (ROI; 32 × 32 pixels), at the end of the surgical instrument on the video images, as supervised data. We applied data augmentation to these ROIs. We employed a You Only Look Once Version 2 (YOLOv2) -based convolutional neural network as the network model for training. The detectors were validated to evaluate average detection precision. The proposed algorithm used the central coordinates of the bounding boxes predicted by YOLOv2. Using the test data, we calculated the detection rate. The average precision (AP) for the ROIs, without data augmentation, was 0.4272 ± 0.108. The AP with data augmentation, of 0.7718 ± 0.0824, was significantly higher than that without data augmentation. The detection rates, including the calculated coordinates of the center points in the centers of 8 × 8 pixels and 16 × 16 pixels, were 0.6100 ± 0.1014 and 0.9653 ± 0.0177, respectively. We expect that the proposed algorithm will be efficient for the analysis of surgical records.

[1]  G. Sutherland,et al.  Forces of Tool-Tissue Interaction to Assess Surgical Skill Level , 2017, JAMA surgery.

[2]  M. Jorge Cardoso,et al.  Improving Data Augmentation for Medical Image Segmentation , 2018 .

[3]  Xiaohong W. Gao,et al.  Classification of CT brain images based on deep learning networks , 2017, Comput. Methods Programs Biomed..

[4]  Guowu Yang,et al.  An experimental study on breast lesion detection and classification from ultrasound images using deep learning architectures , 2019, BMC Medical Imaging.

[5]  Kevin M. Cury,et al.  DeepLabCut: markerless pose estimation of user-defined body parts with deep learning , 2018, Nature Neuroscience.

[6]  Qianqian Zhang,et al.  MRI Gibbs‐ringing artifact reduction by means of machine learning using convolutional neural networks , 2019, Magnetic resonance in medicine.

[7]  Michał Grochowski,et al.  Data augmentation for improving deep learning in image classification problem , 2018, 2018 International Interdisciplinary PhD Workshop (IIPhDW).

[8]  Bernt Schiele,et al.  Detecting Surgical Tools by Modelling Local Appearance and Global Shape , 2015, IEEE Transactions on Medical Imaging.

[9]  Geoff Nitschke,et al.  Improving Deep Learning using Generic Data Augmentation , 2017 .

[10]  Nico Karssemeijer,et al.  Using deep learning to segment breast and fibroglandular tissue in MRI volumes , 2017, Medical physics.

[11]  Sandrine Voros,et al.  Real-time tracking of surgical instruments based on spatio-temporal context and deep learning , 2019, Computer assisted surgery.

[12]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[13]  Kota Yokoyama,et al.  Artificial intelligence using neural network architecture for radiology (AINNAR): classification of MR imaging sequences , 2018, Japanese Journal of Radiology.

[14]  John M. Pecarina,et al.  Improved Aircraft Recognition for Aerial Refueling Through Data Augmentation in Convolutional Neural Networks , 2016, ISVC.

[15]  Masashi Kawakami,et al.  Automatic Detection of a Standard Line for Brain Magnetic Resonance Imaging Using Deep Learning , 2019, Applied Sciences.

[16]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Zeng-Guang Hou,et al.  RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[18]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[20]  Qinghua Zhao,et al.  Brain tumor classification for MR images using transfer learning and fine-tuning , 2019, Comput. Medical Imaging Graph..

[21]  J. Wang,et al.  Convolutional Neural Network for Automated FLAIR Lesion Segmentation on Clinical Brain MR Imaging , 2019, American Journal of Neuroradiology.

[22]  K. Houkin,et al.  A Pilot Study on Measuring Tissue Motion During Carotid Surgery Using Video-Based Analyses for the Objective Assessment of Surgical Performance , 2019, World Journal of Surgery.

[23]  Р Ю Чуйков,et al.  Обнаружение транспортных средств на изображениях загородных шоссе на основе метода Single shot multibox Detector , 2017 .

[24]  Hiroyuki Sugimori,et al.  Classification of Computed Tomography Images in Different Slice Positions Using Deep Learning , 2018, Journal of healthcare engineering.

[25]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Hiroyuki Sugimori,et al.  Evaluating the Overall Accuracy of Additional Learning and Automatic Classification System for CT Images , 2019, Applied Sciences.

[27]  J. Birkmeyer,et al.  Surgical skill and complication rates after bariatric surgery. , 2013, The New England journal of medicine.

[28]  O. Abe,et al.  Liver Fibrosis: Deep Convolutional Neural Network for Staging by Using Gadoxetic Acid-enhanced Hepatobiliary Phase MR Images. , 2017, Radiology.

[29]  Liang Chen,et al.  Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks , 2017, NeuroImage: Clinical.

[30]  Mubarak Shah,et al.  ClusterNet: Detecting Small Objects in Large Scenes by Exploiting Spatio-Temporal Information , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Kai Chen,et al.  Optimizing Video Object Detection via a Scale-Time Lattice , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Max A. Viergever,et al.  Automatic brain tissue segmentation in fetal MRI using convolutional neural networks , 2019, Magnetic resonance imaging.

[34]  Lena Maier-Hein,et al.  2017 Robotic Instrument Segmentation Challenge , 2019, ArXiv.

[35]  K. Ogasawara,et al.  Preoperative cervical carotid artery contrast-enhanced ultrasound findings are associated with development of microembolic signals on transcranial Doppler during carotid exposure in endarterectomy. , 2017, Atherosclerosis.

[36]  Faliang Chang,et al.  Real-time surgical instrument detection in robot-assisted surgery using a convolutional neural network cascade , 2019, Healthcare technology letters.

[37]  Jaesoon Choi,et al.  Robust Real-Time Detection of Laparoscopic Instruments in Robot Surgery Using Convolutional Neural Networks with Motion Vector Prediction , 2019, Applied Sciences.

[38]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[39]  Jie Yang,et al.  Deep Learning and Multi-Sensor Fusion for Glioma Classification Using Multistream 2D Convolutional Networks , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[40]  Aymeric Histace,et al.  Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge , 2017, IEEE Transactions on Medical Imaging.

[41]  S. M. Kamrul Hasan,et al.  U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instruments from Laparoscopic Images , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[42]  Ronald M. Summers,et al.  Anatomy-specific classification of medical images using deep convolutional nets , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[43]  Ellen Poliakoff,et al.  Machine learning algorithm validation with a limited sample size , 2019, PloS one.

[44]  Jonathan Tompson,et al.  PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model , 2018, ECCV.

[45]  A. Katsumata,et al.  Automatic detection and classification of radiolucent lesions in the mandible on panoramic radiographs using a deep learning object detection technique. , 2019, Oral surgery, oral medicine, oral pathology and oral radiology.