Review on the Applications of Deep Learning in the Analysis of Gastrointestinal Endoscopy Images

Gastrointestinal (GI) disease is one of the most common diseases and primarily examined by GI endoscopy. Recently, deep learning (DL), in particular convolutional neural networks (CNNs) have made achievements in GI endoscopy image analysis. This review focuses on the applications of DL methods in the analysis of GI images. We summarized and compared the latest published literature related to the common clinical GI diseases and covers the key applications of DL in GI image detection, classification, segmentation, recognition, location, and other tasks. At the end, we give a discussion on the challenges and the research directions of GI image analysis based on DL in the future.

[1]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Stephen Lin,et al.  Automatic Feature Learning to Grade Nuclear Cataracts Based on Deep Learning , 2014, IEEE Transactions on Biomedical Engineering.

[3]  Matthew T. Freedman,et al.  Artificial convolution neural network techniques and applications for lung nodule detection , 1995, IEEE Trans. Medical Imaging.

[4]  George Azzopardi,et al.  A deep learning approach for detecting and correcting highlights in endoscopic images , 2017, 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA).

[5]  Nima Tajbakhsh,et al.  Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[6]  Jie Zheng,et al.  Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process , 2016, Medical Image Anal..

[7]  Henry Horng-Shing Lu,et al.  Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis. , 2017, Gastroenterology.

[8]  H. Duan,et al.  Gastric precancerous diseases classification using CNN with a concise model , 2017, PloS one.

[9]  M. Fujishiro,et al.  Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. , 2019, Gastrointestinal endoscopy.

[10]  Max Q.-H. Meng,et al.  A study on automated segmentation of blood regions in Wireless Capsule Endoscopy images using fully convolutional networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[11]  Aymeric Histace,et al.  Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge , 2017, IEEE Transactions on Medical Imaging.

[12]  M. Fujishiro,et al.  Diagnosis using deep-learning artificial intelligence based on the endocytoscopic observation of the esophagus , 2018, Esophagus.

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  J. Coebergh,et al.  Recent trends in cancer survival across Europe between 2000 and 2004: a model-based period analysis from 12 cancer registries. , 2008, European journal of cancer.

[15]  Fei Gao,et al.  Convolutional neural networks for intestinal hemorrhage detection in wireless capsule endoscopy images , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[16]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[17]  Bing Zeng,et al.  Automatic Detection of Early Gastrointestinal Cancer Lesions Based on Optimal Feature Extraction from Gastroscopic Images , 2015 .

[18]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[19]  PhD Kay Washington MD.  7th Edition of the AJCC Cancer Staging Manual: Stomach , 2010, Annals of Surgical Oncology.

[20]  Jiwen Lu,et al.  PCANet: A Simple Deep Learning Baseline for Image Classification? , 2014, IEEE Transactions on Image Processing.

[21]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[23]  M. Fujishiro,et al.  Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images , 2018, Gastric Cancer.

[24]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[25]  Rong Li,et al.  Gastric Pathology Image Recognition Based on Deep Residual Networks , 2018, 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC).

[26]  D. Iakovidis,et al.  Software for enhanced video capsule endoscopy: challenges for essential progress , 2015, Nature Reviews Gastroenterology &Hepatology.

[27]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.

[28]  K. Koike,et al.  Automatic detection of erosions and ulcerations in wireless capsule endoscopy images based on a deep convolutional neural network. , 2019, Gastrointestinal endoscopy.

[29]  Ramesh Jain,et al.  Hookworm Detection in Wireless Capsule Endoscopy Images With Deep Learning , 2018, IEEE Transactions on Image Processing.

[30]  P. Cattin,et al.  Multi-dimensional Gated Recurrent Units for the Segmentation of Biomedical 3D-Data , 2016, LABELS/DLMIA@MICCAI.

[31]  Satoshi Tanabe,et al.  Diagnosis of Obscure Gastrointestinal Bleeding , 2016, Clinical endoscopy.

[32]  S. Nomura,et al.  Application of Convolutional Neural Networks in the Diagnosis of Helicobacter pylori Infection Based on Endoscopic Images , 2017, EBioMedicine.

[33]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[34]  Nicolas Chapados,et al.  Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model , 2017, Gut.

[35]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[36]  Y. Zhong,et al.  Progress with each passing day: role of endoscopy in early gastric cancer , 2015 .

[37]  Hao Chen,et al.  Integrating Online and Offline Three-Dimensional Deep Learning for Automated Polyp Detection in Colonoscopy Videos , 2017, IEEE Journal of Biomedical and Health Informatics.

[38]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Andrea Asperti,et al.  The Effectiveness of Data Augmentation for Detection of Gastrointestinal Diseases from Endoscopical Images , 2017, BIOIMAGING.

[41]  Takumi Itoh,et al.  Deep learning analyzes Helicobacter pylori infection by upper gastrointestinal endoscopy images , 2018, Endoscopy International Open.

[42]  Mun-Cheon Kang,et al.  A Novel Gastric Ulcer Differentiation System Using Convolutional Neural Networks , 2018, 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS).

[43]  Masahiro Murakawa,et al.  Gastric Pathology Image Classification Using Stepwise Fine-Tuning for Deep Neural Networks , 2018, Journal of healthcare engineering.

[44]  LinLin Shen,et al.  Deep learning based gastric cancer identification , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[45]  Jacob Chakareski,et al.  Effective Deep Learning for Semantic Segmentation Based Bleeding Zone Detection in Capsule Endoscopy Images , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[46]  A. Jemal,et al.  Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries , 2018, CA: a cancer journal for clinicians.

[47]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Tao Gan,et al.  An automatic annotation method for early esophageal cancers based on saliency guided superpixel segmentation , 2017, ICBCI 2017.

[49]  Gregory G. Slabaugh,et al.  Automatic Segmentation of Polyps in Colonoscopic Narrow-Band Imaging Data , 2012, IEEE Transactions on Biomedical Engineering.

[50]  Andreas Uhl,et al.  Colonic Polyp Classification with Convolutional Neural Networks , 2016, 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS).

[51]  Nader Karimi,et al.  Segmentation of Bleeding Regions in Wireless Capsule Endoscopy for Detection of Informative Frames , 2018, Biomed. Signal Process. Control..

[52]  Ilangko Balasingham,et al.  Automatic Colon Polyp Detection Using Region Based Deep CNN and Post Learning Approaches , 2018, IEEE Access.

[53]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[54]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[55]  Takuya Yamada,et al.  Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists. , 2019, Gastrointestinal endoscopy.

[56]  Jin Chen,et al.  A hybrid convolutional neural networks with extreme learning machine for WCE image classification , 2015, 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[57]  Weifeng Liu,et al.  Canonical correlation analysis networks for two-view image recognition , 2017, Inf. Sci..

[58]  S. T. Devarakonda,et al.  A convolutional neural network approach for abnormality detection in Wireless Capsule Endoscopy , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[59]  Xiao Wu,et al.  Automatic content understanding with cascaded spatial-temporal deep framework for capsule endoscopy videos , 2017, Neurocomputing.

[60]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[61]  Carmen C. Y. Poon,et al.  Automatic Detection and Classification of Colorectal Polyps by Transferring Low-Level CNN Features From Nonmedical Domain , 2017, IEEE Journal of Biomedical and Health Informatics.

[62]  Wei-Min Liu,et al.  Semantic Segmentation of Colorectal Polyps with DeepLab and LSTM Networks , 2018, 2018 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW).

[63]  Shangbo Zhou,et al.  Deep Convolutional Neural Networks for WCE Abnormality Detection: CNN Architecture, Region Proposal and Transfer Learning , 2019, IEEE Access.

[64]  J. Saurin,et al.  A neural network algorithm for detection of GI angiectasia during small-bowel capsule endoscopy. , 2019, Gastrointestinal endoscopy.

[65]  Syed Muhammad Anwar,et al.  Deep Learning in Medical Image Analysis , 2017 .

[66]  Jayan Mannath,et al.  Role of endoscopy in early oesophageal cancer , 2016, Nature Reviews Gastroenterology &Hepatology.

[67]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[68]  Max Q.-H. Meng,et al.  Gastrointestinal bleeding detection in wireless capsule endoscopy images using handcrafted and CNN features , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[69]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[70]  Michael Riegler,et al.  Deep learning and handcrafted feature based approaches for automatic detection of angiectasia , 2018, 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[71]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[72]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[73]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[74]  Chong-Wah Ngo,et al.  Automatic Hookworm Detection in Wireless Capsule Endoscopy Images , 2016, IEEE Transactions on Medical Imaging.

[75]  Xiaoqi Liu,et al.  Fine-tuning Pre-trained Convolutional Neural Networks for Gastric Precancerous Disease Classification on Magnification Narrow-band Imaging Images , 2020, Neurocomputing.

[76]  Ilangko Balasingham,et al.  Comparison of hand-craft feature based SVM and CNN based deep learning framework for automatic polyp classification , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[77]  Fons van der Sommen,et al.  Automatic Detection of Early Esophageal Cancer with CNNS Using Transfer Learning , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[78]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Bram van Ginneken,et al.  Fast Convolutional Neural Network Training Using Selective Data Sampling: Application to Hemorrhage Detection in Color Fundus Images , 2016, IEEE Transactions on Medical Imaging.

[80]  Hyunjin Park,et al.  Convolutional neural network classifier for distinguishing Barrett's esophagus and neoplasia endomicroscopy images , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[81]  Qian Li,et al.  Annotating Early Esophageal Cancers Based on Two Saliency Levels of Gastroscopic Images , 2018, Journal of Medical Systems.

[82]  Jing Cheng,et al.  Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy. , 2019, Gastrointestinal endoscopy.

[83]  Guoqiang Han,et al.  Quantitative analysis of patients with celiac disease by video capsule endoscopy: A deep learning method , 2017, Comput. Biol. Medicine.

[84]  Lipo Wang,et al.  Deep Learning Applications in Medical Image Analysis , 2018, IEEE Access.

[85]  Dimitris K. Iakovidis,et al.  Detecting and Locating Gastrointestinal Anomalies Using Deep Learning and Iterative Cluster Unification , 2018, IEEE Transactions on Medical Imaging.

[86]  Jordi Vitrià,et al.  Generic Feature Learning for Wireless Capsule Endoscopy Analysis , 2016, Comput. Biol. Medicine.

[87]  Andreas Uhl,et al.  Exploring Deep Learning and Transfer Learning for Colonic Polyp Classification , 2016, Comput. Math. Methods Medicine.

[88]  Yao Hu,et al.  Transfer Learning with Convolutional Neural Network for Early Gastric Cancer Classification on Magnifiying Narrow-Band Imaging Images , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).