Proposing novel methods for gynecologic surgical action recognition on laparoscopic videos

Laparoscopy or minimally-invasive surgery (MIS) is performed by inserting a camera called endoscope inside the body to display the surgical actions online with the ability to record and archive the video. Recognizing the surgical actions automatically from the laparoscopic videos have many applications such as surgical skill assessment, teaching purposes, and workflow recognition but is a challenging task. The main aim of this study is proposing novel automatic methods for surgical action recognition from the laparoscopic video frames. For this purpose, three different scenarios are designed, evaluated and compared using 5-fold cross validation strategy. The first and the second scenarios are based on deep neural networks and combination of pre-trained CNNs and conventional machine learning models, respectively. The last scenario combines handcraft feature extraction, pre-trained CNNs, feature engineering based on complex networks and conventional classifiers. Dataset analyzed in this study is ITEC LapGyn4 Gynecologic Laparoscopy Image dataset. Experimental results show that the second and the third scenarios have highly desirable performance for multi-instance surgical action recognition with the average accuracy of 99.20 and AUC of 99.12. On the other hand, for single-instance surgical action recognition, the third scenario outperforms the compared ones with the average accuracy of 99.05 and AUC of 96.41. Moreover, different feature sets in the third scenario are ranked and assigned the importance score based on “Mean Decrease of Accuracy” measure. The first-ranked features are the deep features extracted from our proposed CNNs in the first scenario and the second-ranked ones are the features engineered from the complex networks.

[1]  Jing Wang,et al.  Endoscope localization and gastrointestinal feature map construction based on monocular SLAM technology. , 2019, Journal of infection and public health.

[2]  A. Helen Victoria,et al.  Automatic tuning of hyperparameters using Bayesian optimization , 2020, Evolving Systems.

[3]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Constantinos G. Loukas,et al.  Keyframe extraction from laparoscopic videos based on visual saliency detection , 2018, Comput. Methods Programs Biomed..

[5]  Fan Yang,et al.  Endoscopic video defogging using luminance blending , 2019, Healthcare technology letters.

[6]  Joel J. P. C. Rodrigues,et al.  A novel deep learning based framework for the detection and classification of breast cancer using transfer learning , 2019, Pattern Recognit. Lett..

[7]  Romuere Rôdrigues Veloso e Silva,et al.  Leukemia diagnosis in blood slides using transfer learning in CNNs and SVM for classification , 2018, Eng. Appl. Artif. Intell..

[8]  Karel J. Zuiderveld,et al.  Contrast Limited Adaptive Histogram Equalization , 1994, Graphics Gems.

[9]  Jenny Benois-Pineau,et al.  Early and Late Fusion of Temporal Information for Classification of Surgical Actions in Laparoscopic Gynecology , 2018, 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS).

[10]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[11]  Abeer Alsadoon,et al.  A novel enhanced hybrid recursive algorithm: Image processing based augmented reality for gallbladder and uterus visualisation , 2020 .

[12]  Lena Maier-Hein,et al.  Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy , 2017, IEEE Transactions on Biomedical Engineering.

[13]  Andru Putra Twinanda,et al.  EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos , 2016, IEEE Transactions on Medical Imaging.

[14]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[15]  Anupam Joshi,et al.  Detection of unsafe action from laparoscopic cholecystectomy video , 2012, IHI '12.

[16]  Philippe Cinquin,et al.  Automatic Detection of Instruments in Laparoscopic Images: A First Step Towards High-level Command of Robotic Endoscopic Holders , 2007, The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006..

[17]  Gaurav Yengera,et al.  Future-State Predicting LSTM for Early Surgery Type Recognition , 2018, IEEE Transactions on Medical Imaging.

[18]  Philip Kollmannsberger,et al.  Efficient Classification of White Blood Cell Leukemia with Improved Swarm Optimization of Deep Features , 2020, Scientific Reports.

[19]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[20]  A. Buia,et al.  Laparoscopic surgery: A qualified systematic review. , 2015, World journal of methodology.

[21]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[22]  Pablo Lamata,et al.  Laparoscopic video analysis for training and image-guided surgery , 2011, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[23]  Hao Chen,et al.  Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis , 2019, Medical Image Anal..

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Bin Sheng,et al.  Computer-Assisted Decision Support System in Pulmonary Cancer detection and stage classification on CT images , 2018, J. Biomed. Informatics.

[26]  J. A. Sánchez-Margallo,et al.  EVA: Laparoscopic Instrument Tracking Based on Endoscopic Video Analysis for Psychomotor Skills Assessment , 2013, Surgical Endoscopy.

[27]  Shahab Abdulla,et al.  EEG sleep stages identification based on weighted undirected complex networks , 2020, Comput. Methods Programs Biomed..

[28]  Roberto A. Hexsel,et al.  Particle filtering in the Hough space for instrument tracking , 2012, Comput. Biol. Medicine.

[29]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[30]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[31]  Takahiro Yamanashi,et al.  Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach , 2019, Surgical Endoscopy.

[32]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Tardi Tjahjadi,et al.  Fusing dynamic deep learned features and handcrafted features for facial expression recognition , 2019, J. Vis. Commun. Image Represent..

[34]  Alex Bavelas.  A Mathematical Model for Group Structures , 1948 .

[35]  V. Latora,et al.  Complex Networks: Principles, Methods and Applications , 2017 .

[36]  Sandro Sperandei,et al.  Understanding logistic regression analysis , 2014, Biochemia medica.

[37]  Carlos S. Lima,et al.  Wireless capsule endoscopic frame classification scheme based on higher order statistics of multi-scale texture descriptors , 2009 .

[38]  Jiawei Han,et al.  Data Mining: Concepts and Techniques , 2000 .

[39]  Klaus Schöffmann,et al.  Lapgyn4: a dataset for 4 automatic content analysis problems in the domain of laparoscopic gynecology , 2018, MMSys.

[40]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[41]  Congcong Wang,et al.  A Smoke Removal Method for Laparoscopic Images , 2018, ArXiv.

[42]  Klaus Schöffmann,et al.  Learning laparoscopic video shot classification for gynecological surgery , 2018, Multimedia Tools and Applications.

[43]  N. Yadav,et al.  Minimally invasive (endoscopic-computer assisted) surgery: Technique and review , 2016, Annals of maxillofacial surgery.

[44]  Ryszard Klempous,et al.  Application of Image Processing and Virtual Reality Technologies in Simulation of Laparoscopic Procedures , 2015, EUROCAST.