Detecting, Localising and Classifying Polyps from Colonoscopy Videos using Deep Learning

In this paper, we propose and analyse a system that can automatically detect, localise and classify polyps from colonoscopy videos. The detection of frames with polyps is formulated as a few-shot anomaly classification problem, where the training set is highly imbalanced with the large majority of frames consisting of normal images and a small minority comprising frames with polyps. Colonoscopy videos may contain blurry images and frames displaying feces and water jet sprays to clean the colon – such frames can mistakenly be detected as anomalies, so we have implemented a classifier to reject these two types of frames before polyp detection takes place. Next, given a frame containing a polyp, our method localises (with a bounding box around the polyp) and classifies it into five different classes. Furthermore, we study a method to improve the reliability and interpretability of the classification result using uncertainty estimation and classification calibration. Classification uncertainty and calibration not only help improve classification accuracy by rejecting lowconfidence and high-uncertain results, but can be used by doctors to decide how to decide on the classification of a polyp. All the proposed detection, localisation and classification methods are tested using large data sets and compared with relevant baseline approaches.

[1]  Jihoon Kim,et al.  Calibrating predictive model estimates to support personalized medicine , 2011, J. Am. Medical Informatics Assoc..

[2]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Deng Cai,et al.  Laplacian Score for Feature Selection , 2005, NIPS.

[4]  Doina Precup,et al.  Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation , 2018, MICCAI.

[5]  Konstantinos Kamnitsas,et al.  Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation , 2019, MICCAI.

[6]  Ling Chen,et al.  Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection , 2018, KDD.

[7]  David Lieberman,et al.  Colorectal Cancer Screening: Recommendations for Physicians and Patients from the U.S. Multi-Society Task Force on Colorectal Cancer , 2017, The American Journal of Gastroenterology.

[8]  Ruslan Salakhutdinov,et al.  A Simple Approach to the Noisy Label Problem Through the Gambler's Loss , 2019 .

[9]  A. Jemal,et al.  Colorectal cancer statistics, 2014 , 2014, CA: a cancer journal for clinicians.

[10]  M. Kudo,et al.  Computer-Aided Diagnosis Based on Convolutional Neural Network System for Colorectal Polyp Classification: Preliminary Experience , 2017, Oncology.

[11]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[12]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[13]  Carl Doersch,et al.  Tutorial on Variational Autoencoders , 2016, ArXiv.

[14]  P. Bossuyt,et al.  Polyp Miss Rate Determined by Tandem Colonoscopy: A Systematic Review , 2006, The American Journal of Gastroenterology.

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Victor S. Lempitsky,et al.  Unsupervised Domain Adaptation by Backpropagation , 2014, ICML.

[17]  Ramesh Nallapati,et al.  OCGAN: One-Class Novelty Detection Using GANs With Constrained Latent Representations , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Noriya Uedo,et al.  Narrow‐band imaging with dual focus magnification in differentiating colorectal neoplasia , 2013, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[19]  Gustavo Carneiro,et al.  Few-Shot Anomaly Detection for Polyp Frames from Colonoscopy , 2020, MICCAI.

[20]  Longbing Cao,et al.  Deep Learning for Anomaly Detection: A Review , 2020, ArXiv.

[21]  Michael I. Jordan,et al.  Bayesian parameter estimation via variational methods , 2000, Stat. Comput..

[22]  Gustavo Carneiro,et al.  Deep learning uncertainty and confidence calibration for the five-class polyp classification from colonoscopy , 2020, Medical Image Anal..

[23]  Alex Kendall,et al.  Concrete Dropout , 2017, NIPS.

[24]  S. E. Ahmed,et al.  Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference , 2008, Technometrics.

[25]  Hayit Greenspan,et al.  Chest pathology detection using deep learning with non-medical training , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[26]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[27]  Saeed Hassanpour,et al.  Deep Learning for Classification of Colorectal Polyps on Whole-slide Images , 2017, Journal of pathology informatics.

[28]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[30]  Joseph Bullock,et al.  XNet: a convolutional neural network (CNN) implementation for medical x-ray image segmentation suitable for small datasets , 2018, Medical Imaging.

[31]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[32]  Ahmedin Jemal,et al.  Global patterns and trends in colorectal cancer incidence and mortality , 2016, Gut.

[33]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[34]  Aymeric Histace,et al.  Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge , 2017, IEEE Transactions on Medical Imaging.

[35]  Shoichi Saito,et al.  Validation study for development of the Japan NBI Expert Team classification of colorectal lesions , 2018, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[36]  Jürgen Schmidhuber,et al.  Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction , 2011, ICANN.

[37]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Bin Yang,et al.  Learning to Reweight Examples for Robust Deep Learning , 2018, ICML.

[39]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[40]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[41]  Yongxin Yang,et al.  Learning to Generalize: Meta-Learning for Domain Generalization , 2017, AAAI.

[42]  Anton van den Hengel,et al.  Deep Anomaly Detection with Deviation Networks , 2019, KDD.

[43]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[45]  Bo Zong,et al.  Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection , 2018, ICLR.

[46]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[47]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[48]  Gustavo Carneiro,et al.  Computer-aided diagnosis for characterisation of colorectal lesions: a comprehensive software including serrated lesions. , 2020, Gastrointestinal endoscopy.

[49]  Gustavo Carneiro,et al.  Sa1908 COMPUTER-AIDED DIAGNOSIS FOR CHARACTERISING COLORECTAL LESIONS: INTERIM RESULTS OF A NEWLY DEVELOPED SOFTWARE , 2018, Gastrointestinal Endoscopy.

[50]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[51]  M. Jorge Cardoso,et al.  Towards safe deep learning: accurately quantifying biomarker uncertainty in neural network predictions , 2018, MICCAI.

[52]  Shinji Tanaka,et al.  Validation of a simple classification system for endoscopic diagnosis of small colorectal polyps using narrow-band imaging. , 2012, Gastroenterology.

[53]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[54]  Shinji Tanaka,et al.  Exploring texture Transfer Learning for Colonic Polyp Classification via Convolutional Neural Networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[55]  Svetha Venkatesh,et al.  Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[56]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[57]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[58]  B. Levin,et al.  Screening and surveillance for the early detection of colorectal cancer and adenomatous polyps, 2008: a joint guideline from the American Cancer Society, the US Multi-Society Task Force on Colorectal Cancer, and the American College of Radiology. , 2008, Gastroenterology.

[59]  Yuyuan Liu,et al.  Photoshopping Colonoscopy Video Frames , 2019, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[60]  Georg Langs,et al.  f‐AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks , 2019, Medical Image Anal..

[61]  Shinji Tanaka,et al.  Endoscopic prediction of deep submucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classification. , 2013, Gastrointestinal endoscopy.