Depth estimation from single-shot monocular endoscope image using image domain adaptation and edge-aware depth estimation

This paper proposes a depth estimation method from a single-shot monocular endoscopic image. Automated understanding of endoscopic images is important for diagnosis and treatment assistance. Not only the images themselves but also depth information about the images help make the understanding of endoscopic images, such as lesion-size measurements, more accurate. Previous depth estimation methods have used stereo cameras or time-series images. However, many endoscope imaging systems do not support the use of stereo endoscopes and video capturing. Also, automatic classification or recognition of large number of previously stored single-shot monocular endoscopic images is required to perform retrospective studies of endoscopic image analysis. We propose a depth estimation method from a single-shot monocular endoscopic image using Lambertian surface translation by domain adaptation and depth estimation using multi-scale edge loss. The difficulty of the depth estimation is that we cannot obtain real endoscopic images and their corresponding depth images. Depth sensors cannot be attached to endoscopes because of the size limitation. To tackle the difficulty, we employ a two-step estimation process including Lambertian surface translation from unpaired data and depth estimation. The texture and specular reflection on the surface of an organ reduce the accuracy of depth estimations. We apply Lambertian surface translation to an endoscopic image to remove these texture and reflections. Then, we estimate the depth by using a fully convolutional network (FCN). During the training of the FCN, improvement of the object edge similarity between an estimated image and a ground truth depth image is important for getting better results. We introduced a muti-scale edge loss function to improve the accuracy of depth estimation. We quantitatively evaluated the proposed method using real colonoscopic images. The estimated depth values were proportional to the real depth values. Furthermore, we applied the estimated depth images to automated anatomical location identification of colonoscopic images using a convolutional neural network. The identification accuracy of the network improved from 69.2% to 74.1% by using the estimated depth images.