Polyp segmentation and classification using predicted depth from monocular endoscopy

Colorectal cancer is the fourth leading cause of cancer deaths worldwide, the standard for detection and prevention is the identification and removal of premalignant lesions through optical colonoscopy. More than 60% of colorectal cancer cases are attributed to missed polyps. Current procedures for automated polyp detection are limited by the amount of data available for training, underrepresentation of non-polypoid lesions and lesions which are inherently difficult to label and do not incorporate information about the topography of the surface of the lumen. It has been shown that information related to depth and topography of the surface of the lumen can boost subjective lesion detection. In this work, we add predicted depth information as an additional mode of data when training deep networks for polyp detection, segmentation and classification. We use conditional GANs to predict depth from monocular endoscopy images and fuse these predicted depth maps with RGB white light images in feature space. Our empirical analysis demonstrates that we achieve state-of-the-art results with RGB-D polyp segmentation with a 98% accuracy on four different publically available datasets. Moreover, we demonstrate a 87.24% accuracy on lesion classification. We also show that our networks can domain adapt to a variety of different kinds of data from different sources.

[1]  Ian D. Reid,et al.  Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Daniel Pizarro-Perez,et al.  Computer-Aided Classification of Gastrointestinal Lesions in Regular Colonoscopy , 2016, IEEE Transactions on Medical Imaging.

[3]  Faisal Mahmood,et al.  Multimodal Densenet , 2018, ArXiv.

[4]  Alexei A. Efros,et al.  Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics , 2010, ECCV.

[5]  Fei Luo,et al.  RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation , 2018, ArXiv.

[6]  Sandra Sudarsky,et al.  Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images , 2018, Physics in medicine and biology.

[7]  Daniel Cremers,et al.  FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture , 2016, ACCV.

[8]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[9]  R. Pearson.  Association of Colonoscopy and Death from Colorectal Cancer , 2009 .

[10]  S. Winawer,et al.  The History of Colorectal Cancer Screening: A Personal Perspective , 2015, Digestive Diseases and Sciences.

[11]  Faisal Mahmood,et al.  Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training , 2017, IEEE Transactions on Medical Imaging.

[12]  Faisal Mahmood,et al.  Topographical reconstructions from monocular optical colonoscopy images via deep learning , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[13]  Takeo Kanade,et al.  Estimating Spatial Layout of Rooms using Volumetric Reasoning about Objects and Surfaces , 2010, NIPS.

[14]  D. Ransohoff,et al.  How Much Does Colonoscopy Reduce Colon Cancer Mortality? , 2009, Annals of Internal Medicine.

[15]  Nima Tajbakhsh,et al.  Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information , 2016, IEEE Transactions on Medical Imaging.

[16]  Faisal Mahmood,et al.  Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy , 2017, Medical Image Anal..

[17]  A. Zauber,et al.  Prevention of colorectal cancer by colonoscopic polypectomy. The National Polyp Study Workgroup. , 1993 .

[18]  Fernando Vilariño,et al.  Towards automatic polyp detection with a polyp appearance model , 2012, Pattern Recognit..

[19]  Faisal Mahmood,et al.  Deep learning-based depth estimation from a synthetic endoscopy image training set , 2018, Medical Imaging.

[20]  David A. Forsyth,et al.  Thinking Inside the Box: Using Appearance Models and Context Based on Room Geometry , 2010, ECCV.

[21]  Aymeric Histace,et al.  Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer , 2014, International Journal of Computer Assisted Radiology and Surgery.

[22]  Bernt Schiele,et al.  Disparity statistics for pedestrian detection: combining appearance, motion and stereo , 2010, ECCV 2010.