Whole Stomach 3D Reconstruction and Frame Localization From Monocular Endoscope Video

Gastric endoscopy is a common clinical practice that enables medical doctors to diagnose various lesions inside a stomach. In order to identify the location of a gastric lesion such as early cancer and a peptic ulcer within the stomach, this work addresses to reconstruct the color-textured 3D model of a whole stomach from a standard monocular endoscope video and localize any selected video frame to the 3D model. We examine how to enable structure-from-motion (SfM) to reconstruct the whole shape of a stomach from endoscope images, which is a challenging task due to the texture-less nature of the stomach surface. We specifically investigate the combined effect of chromo-endoscopy and color channel selection on SfM to increase the number of feature points. We also design a plane fitting-based algorithm for 3D point outliers removal to improve the 3D model quality. We show that whole stomach 3D reconstruction can be achieved (more than 90% of the frames can be reconstructed) by using red channel images captured under chromo-endoscopy by spreading indigo carmine (IC) dye on the stomach surface. In experimental results, we demonstrate the reconstructed 3D models for seven subjects and the application of lesion localization and reconstruction. The methodology and results presented in this paper could offer some valuable reference to other researchers and also could be an excellent tool for gastric surgeons in various computer-aided diagnosis applications.

[1]  Paolo Cignoni,et al.  MeshLab: an Open-Source Mesh Processing Tool , 2008, Eurographics Italian Chapter Conference.

[2]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[4]  Alexandre Hostettler,et al.  Live Tracking and Dense Reconstruction for Handheld Monocular Endoscopy , 2019, IEEE Transactions on Medical Imaging.

[5]  Masatoshi Okutomi,et al.  3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[6]  Maks Ovsjanikov,et al.  PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds , 2019, Comput. Graph. Forum.

[7]  Long Chen,et al.  SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality , 2018, Comput. Methods Programs Biomed..

[8]  Juho Kannala,et al.  A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[10]  Roland Angst,et al.  3D reconstruction of cystoscopy videos for comprehensive bladder records. , 2017, Biomedical optics express.

[11]  Guang-Zhong Yang,et al.  Enhanced visualisation for minimally invasive surgery , 2012, International Journal of Computer Assisted Radiology and Surgery.

[12]  Armin Schneider,et al.  Time-of-Flight 3-D Endoscopy , 2009, MICCAI.

[13]  Guang-Zhong Yang,et al.  Three-Dimensional Tissue Deformation Recovery and Tracking , 2010, IEEE Signal Processing Magazine.

[14]  Adrien Bartoli,et al.  Enhanced imaging colonoscopy facilitates dense motion-based 3D reconstruction , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[15]  Yosuke Tsuji,et al.  Comparative analysis of upper gastrointestinal endoscopy, double-contrast upper gastrointestinal barium X-ray radiography, and the titer of serum anti-Helicobacter pylori IgG focusing on the diagnosis of atrophic gastritis , 2016, Gastric Cancer.

[16]  Mark Pauly,et al.  Point primitives for interactive modeling and processing of 3D-geometry , 2003 .

[17]  Yuncheng You,et al.  Video‐based 3D reconstruction, laparoscope localization and deformation recovery for abdominal minimally invasive surgery: a survey , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[18]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[19]  Changchang Wu,et al.  Towards Linear-Time Incremental Structure from Motion , 2013, 2013 International Conference on 3D Vision.

[20]  Bing Zeng,et al.  Shape Recovery of Endoscopic Videos by Shape from Shading Using Mesh Regularization , 2017, ICIG.

[21]  Michael Figl,et al.  Reconstruction of a 3D surface from video that is robust to missing data and outliers: Application to minimally invasive surgery using stereo and mono endoscopes , 2012, Medical Image Anal..

[22]  Danail Stoyanov,et al.  Robust Real-Time Visual Odometry for Stereo Endoscopy Using Dense Quadrifocal Tracking , 2014, IPCAI.

[23]  Huilong Duan,et al.  Surface Reconstruction from Tracked Endoscopic Video Using the Structure from Motion Approach , 2013, AE-CAI.

[24]  K. Deguchi,et al.  Shape reconstruction from an endoscope image by shape-from-shading technique for a point light source at the projection center , 1996, Proceedings of the Workshop on Mathematical Methods in Biomedical Image Analysis.

[25]  J. M. M. Montiel,et al.  Visual SLAM for Handheld Monocular Endoscope , 2014, IEEE Transactions on Medical Imaging.

[26]  Yasushi Sano,et al.  American Gastroenterological Association (AGA) Institute technology assessment on image-enhanced endoscopy. , 2008, Gastroenterology.

[27]  Josef Sivic,et al.  NetVLAD: CNN Architecture for Weakly Supervised Place Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[29]  Lena Maier-Hein,et al.  Comparative Validation of Single-Shot Optical Techniques for Laparoscopic 3-D Surface Reconstruction , 2014, IEEE Transactions on Medical Imaging.

[30]  Heoung Keun Kang,et al.  The Role of Three-Dimensional Multidetector CT Gastrography in the Preoperative Imaging of Stomach Cancer: Emphasis on Detection and Localization of the Tumor , 2015, Korean journal of radiology.

[31]  Shinji Tanaka,et al.  Shape Acquisition and Registration for 3D Endoscope Based on Grid Pattern Projection , 2016, ECCV.

[32]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[33]  Christoph Schmalz,et al.  An endoscopic 3D scanner based on structured light , 2012, Medical Image Anal..

[34]  Jason Geng,et al.  Review of 3-D Endoscopic Surface Imaging Techniques , 2014, IEEE Sensors Journal.

[35]  Steven Mills,et al.  Hierarchical Structure from Motion from Endoscopic Video , 2014, IVCNZ '14.

[36]  Danail Stoyanov,et al.  Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy , 2019, International Journal of Computer Assisted Radiology and Surgery.

[37]  Carlos Henrique Quartucci Forster,et al.  Towards 3D reconstruction of endoscope images using shape from shading , 2000, Proceedings 13th Brazilian Symposium on Computer Graphics and Image Processing (Cat. No.PR00878).

[38]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.