Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy

HighlightsSynthetically generated endoscopy data with ground truth depth.An efficient joint CNN‐CRF‐based depth estimation architecture trained from synthetic endoscopy data.Adversarial training for adapting the network trained on synthetic data to real data.State‐of‐the‐art endoscopy depth estimation performance.Validation using registered views of CT and endoscopy data from a porcine colon. Graphical abstract Figure. No caption available. ABSTRACT Colorectal cancer is the fourth leading cause of cancer deaths worldwide and the second leading cause in the United States. The risk of colorectal cancer can be mitigated by the identification and removal of premalignant lesions through optical colonoscopy. Unfortunately, conventional colonoscopy misses more than 20% of the polyps that should be removed, due in part to poor contrast of lesion topography. Imaging depth and tissue topography during a colonoscopy is difficult because of the size constraints of the endoscope and the deforming mucosa. Most existing methods make unrealistic assumptions which limits accuracy and sensitivity. In this paper, we present a method that avoids these restrictions, using a joint deep convolutional neural network‐conditional random field (CNN‐CRF) framework for monocular endoscopy depth estimation. Estimated depth is used to reconstruct the topography of the surface of the colon from a single image. We train the unary and pairwise potential functions of a CRF in a CNN on synthetic data, generated by developing an endoscope camera model and rendering over 200,000 images of an anatomically‐realistic colon.We validate our approach with real endoscopy images from a porcine colon, transferred to a synthetic‐like domain via adversarial training, with ground truth from registered computed tomography measurements. The CNN‐CRF approach estimates depths with a relative error of 0.152 for synthetic endoscopy images and 0.242 for real endoscopy images. We show that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images. This approach can easily be integrated into existing endoscopy systems and provides a foundation for improving computer‐aided detection algorithms for detection, segmentation and classification of lesions.