Improving Colonoscopy Lesion Classification Using Semi-Supervised Deep Learning

While data-driven approaches excel at many image analysis tasks, the performance of these approaches is often limited by a shortage of annotated data available for training. Recent work in semi-supervised learning has shown that meaningful representations of images can be obtained from training with large quantities of unlabeled data, and that these representations can improve the performance of supervised tasks. Here, we demonstrate that an unsupervised jigsaw learning task, in combination with supervised training, results in up to a 9.8% improvement in correctly classifying lesions in colonoscopy images when compared to a fully-supervised baseline. We additionally benchmark improvements in domain adaptation and out-of-distribution detection, and demonstrate that semi-supervised learning outperforms supervised learning in both cases. In colonoscopy applications, these metrics are important given the skill required for endoscopic assessment of lesions, the wide variety of endoscopy systems in use, and the homogeneity that is typical of labeled datasets.

[1]  Tassilo Klein,et al.  Multimodal Self-Supervised Learning for Medical Image Analysis , 2019, IPMI.

[2]  Julien Mairal,et al.  Unsupervised Learning of Visual Features by Contrasting Cluster Assignments , 2020, NeurIPS.

[3]  Silvia Sanduleanu,et al.  Development and validation of the WASP classification system for optical diagnosis of adenomas, hyperplastic polyps and sessile serrated adenomas/polyps , 2015, Gut.

[4]  Thomas G. Dietterich,et al.  Deep Anomaly Detection with Outlier Exposure , 2018, ICLR.

[5]  Jun Ki Min,et al.  Overview of Deep Learning in Gastrointestinal Endoscopy , 2019, Gut and liver.

[6]  Stefan Bauer,et al.  Patient-Specific Semi-supervised Learning for Postoperative Brain Tumor Segmentation , 2014, MICCAI.

[7]  Matthijs Douze,et al.  Deep Clustering for Unsupervised Learning of Visual Features , 2018, ECCV.

[8]  Faisal Mahmood,et al.  Multimodal Densenet , 2018, ArXiv.

[9]  Yuan Zhang,et al.  FocalMix: Semi-Supervised Learning for 3D Medical Image Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yingli Tian,et al.  Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Paolo Favaro,et al.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles , 2016, ECCV.

[12]  Nassir Navab,et al.  Fusing Unsupervised and Supervised Deep Learning for White Matter Lesion Segmentation , 2019, MIDL.

[13]  Fabio Maria Carlucci,et al.  Domain Generalization by Solving Jigsaw Puzzles , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Marius Leordeanu,et al.  Unsupervised Learning from Video to Detect Foreground Objects in Single Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  T. Lui,et al.  Endoscopic prediction of deeply submucosal invasive carcinoma with use of artificial intelligence , 2019, Endoscopy International Open.

[17]  Kevin Gimpel,et al.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks , 2016, ICLR.

[18]  Irfan A. Essa,et al.  Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[19]  Lorenzo Torresani,et al.  Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization , 2018, NeurIPS.

[20]  Shinji Tanaka,et al.  Endoscopic prediction of deep submucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classification. , 2013, Gastrointestinal endoscopy.

[21]  Imari Sato,et al.  Semi-supervised Learning for Biomedical Image Segmentation via Forest Oriented Super Pixels(Voxels) , 2017, MICCAI.

[22]  Alexei A. Efros,et al.  Unsupervised Visual Representation Learning by Context Prediction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[23]  Alejandro F. Frangi,et al.  Retinal Image Synthesis and Semi-Supervised Learning for Glaucoma Assessment , 2019, IEEE Transactions on Medical Imaging.

[24]  Henry Horng-Shing Lu,et al.  Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis. , 2017, Gastroenterology.

[25]  Dawn Song,et al.  Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty , 2019, NeurIPS.

[26]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  X. Zuo,et al.  Kudo's pit pattern classification for colorectal neoplasms: a meta-analysis. , 2014, World journal of gastroenterology.

[28]  Trevor Darrell,et al.  Learning Features by Watching Objects Move , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Paolo Favaro,et al.  Boosting Self-Supervised Learning via Knowledge Transfer , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Georg Langs,et al.  A spatio-temporal latent atlas for semi-supervised learning of fetal brain segmentations and morphological age estimation , 2014, Medical Image Anal..

[31]  S Spoto,et al.  [Colorectal polyps]. , 2003, La Clinica terapeutica.

[32]  Kai Ma,et al.  Revisiting Rubik's Cube: Self-supervised Learning with Volume-wise Transformation for 3D Medical Image Segmentation , 2020, MICCAI.

[33]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[34]  Yong Jae Lee,et al.  Cross-Domain Self-Supervised Multi-task Feature Learning Using Synthetic Imagery , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Nicolas Chapados,et al.  Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model , 2017, Gut.

[36]  A. Jemal,et al.  Colorectal cancer statistics, 2020 , 2020, CA: a cancer journal for clinicians.

[37]  Ben Glocker,et al.  Semi-supervised Learning for Network-Based Cardiac MR Image Segmentation , 2017, MICCAI.

[38]  U. Bagci,et al.  Su1741 COLORECTAL POLYP DIAGNOSIS WITH CONTEMPORARY ARTIFICIAL INTELLIGENCE , 2019, Gastrointestinal Endoscopy.

[39]  Danail Stoyanov,et al.  Artificial intelligence and computer-aided diagnosis in colonoscopy: current evidence and future directions. , 2019, The lancet. Gastroenterology & hepatology.

[40]  Antonio Torralba,et al.  Generating Videos with Scene Dynamics , 2016, NIPS.

[41]  Richard J. Chen,et al.  Polyp segmentation and classification using predicted depth from monocular endoscopy , 2019, Medical Imaging.

[42]  Daniel Pizarro-Perez,et al.  Computer-Aided Classification of Gastrointestinal Lesions in Regular Colonoscopy , 2016, IEEE Transactions on Medical Imaging.

[43]  Sajjad Waheed,et al.  An Automatic Gastrointestinal Polyp Detection System in Video Endoscopy Using Fusion of Color Wavelet and Convolutional Neural Network Features , 2017, Int. J. Biomed. Imaging.

[44]  Florentino Fernández Riverola,et al.  Deep Neural Networks approaches for detecting and classifying colorectal polyps , 2021, Neurocomputing.

[45]  James M. Rehg,et al.  Unsupervised Learning of Edges , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Ann G Zauber,et al.  Colorectal cancer screening: Estimated future colonoscopy need and current volume and capacity , 2016, Cancer.

[47]  Andrew Zisserman,et al.  Look, Listen and Learn , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Josien P. W. Pluim,et al.  Not‐so‐supervised: A survey of semi‐supervised, multi‐instance, and transfer learning in medical image analysis , 2018, Medical Image Anal..

[49]  Carmen C. Y. Poon,et al.  Automatic Detection and Classification of Colorectal Polyps by Transferring Low-Level CNN Features From Nonmedical Domain , 2017, IEEE Journal of Biomedical and Health Informatics.

[50]  Paul M. Thompson,et al.  Agreement-Based Semi-supervised Learning for Skull Stripping , 2010, MICCAI.

[51]  Kaiming He,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  C. Hassan,et al.  Advances, problems, and complications of polypectomy , 2014, Clinical and experimental gastroenterology.

[53]  Klaus H. Maier-Hein,et al.  Exploiting the potential of unlabeled endoscopic video data with self-supervised learning , 2017, International Journal of Computer Assisted Radiology and Surgery.

[54]  Junnan Li,et al.  Improving out-of-distribution generalization via multi-task self-supervised pretraining , 2020, ArXiv.

[55]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[56]  Yujiu Yang,et al.  Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cube , 2019, MICCAI.

[57]  Nikos Komodakis,et al.  Unsupervised Representation Learning by Predicting Image Rotations , 2018, ICLR.

[58]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[59]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.