Sparse-then-dense alignment-based 3D map reconstruction method for endoscopic capsule robots

Despite significant progress achieved in the last decade to convert passive capsule endoscopes to actively controllable robots, robotic capsule endoscopy still has some challenges. In particular, a fully dense three-dimensional (3D) map reconstruction of the explored organ remains an unsolved problem. Such a dense map would help doctors detect the locations and sizes of the diseased areas more reliably, resulting in more accurate diagnoses. In this study, we propose a comprehensive medical 3D reconstruction method for endoscopic capsule robots, which is built in a modular fashion including preprocessing, keyframe selection, sparse-then-dense alignment-based pose estimation, bundle fusion, and shading-based 3D reconstruction. A detailed quantitative analysis is performed using a non-rigid esophagus gastroduodenoscopy simulator, four different endoscopic cameras, a magnetically activated soft capsule robot, a sub-millimeter precise optical motion tracker, and a fine-scale 3D optical scanner, whereas qualitative ex-vivo experiments are performed on a porcine pig stomach. To the best of our knowledge, this study is the first complete endoscopic 3D map reconstruction approach containing all of the necessary functionalities for a therapeutically relevant 3D map reconstruction.

[1]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[2]  Yasin Almalioglu,et al.  A Deep Learning Based 6 Degree-of-Freedom Localization Method for Endoscopic Capsule Robots , 2017, ArXiv.

[3]  Arie E. Kaufman,et al.  3D Surface Reconstruction from Endoscopic Videos , 2008, Visualization in Medicine and Life Sciences.

[4]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[5]  Helder Araujo,et al.  Magnetic-Visual Sensor Fusion based Medical SLAM for Endoscopic Capsule Robot , 2017 .

[6]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[7]  Guang-Zhong Yang,et al.  Metric depth recovery from monocular images using Shape-from-Shading and specularities , 2012, 2012 19th IEEE International Conference on Image Processing.

[8]  Metin Sitti,et al.  Design and Rolling Locomotion of a Magnetically Actuated Soft Capsule Endoscope , 2012, IEEE Transactions on Robotics.

[9]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[10]  Branislav Jaramaz,et al.  A Multi-Image Shape-from-Shading Framework for Near-Lighting Perspective Endoscopes , 2009, International Journal of Computer Vision.

[11]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[12]  William E. Higgins,et al.  Method for radiometric calibration of an endoscope's camera and light source , 2008, SPIE Medical Imaging.

[13]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[14]  Ève Coste-Manière,et al.  Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery , 2001, Proceedings International Workshop on Medical Imaging and Augmented Reality.

[15]  M. Sitti,et al.  Magnetically Actuated Soft Capsule With the Multimodal Drug Release Function , 2013, IEEE/ASME Transactions on Mechatronics.

[16]  Lakmal Seneviratne,et al.  Vision and inertial-based image mapping for capsule endoscopy , 2015, 2015 International Conference on Information and Communication Technology Research (ICTRC).

[17]  Eric Diller,et al.  Biomedical Applications of Untethered Mobile Milli/Microrobots , 2015, Proceedings of the IEEE.

[18]  Berthold K. P. Horn.  Height and gradient from shading , 1989, International Journal of Computer Vision.

[19]  Olivier D. Faugeras,et al.  Shape From Shading , 2006, Handbook of Mathematical Models in Computer Vision.

[20]  Takayuki Okatani,et al.  Shape Reconstruction from an Endoscope Image by Shape from Shading Technique for a Point Light Source at the Projection Center , 1997, Comput. Vis. Image Underst..

[21]  Helder Araújo,et al.  Six Degree-of-Freedom Localization of Endoscopic Capsule Robots using Recurrent Neural Networks embedded into a Convolutional Neural Network , 2017, ArXiv.

[22]  Guang-Zhong Yang,et al.  Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery , 2010, MICCAI.

[23]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[24]  A. E. Conrady.  Decentred Lens-Systems , 1919 .

[25]  Steven M. Seitz,et al.  Multicore bundle adjustment , 2011, CVPR 2011.

[26]  Adrien Bartoli,et al.  Combining Conformal Deformation and Cook–Torrance Shading for 3-D Reconstruction in Laparoscopy , 2014, IEEE Transactions on Biomedical Engineering.

[27]  Jan-Michael Frahm,et al.  Improving 3D surface reconstruction from endoscopic video via fusion and refined reflectance modeling , 2017, Medical Imaging.

[28]  Helder Araújo,et al.  Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots , 2017, Neurocomputing.

[29]  Adrien Bartoli,et al.  Template-Based Conformal Shape-from-Motion-and-Shading for Laparoscopy , 2012, IPCAI.

[30]  Yasin Almalioglu,et al.  Endo-VMFuseNet: Deep Visual-Magnetic Sensor Fusion Approach for Uncalibrated, Unsynchronized and Asymmetric Endoscopic Capsule Robot Localization Data , 2017, ArXiv.

[31]  Yasin Almalioglu,et al.  Endo-VMFuseNet: A Deep Visual-Magnetic Sensor Fusion Approach for Endoscopic Capsule Robots , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[32]  Arie E. Kaufman,et al.  Depth Reconstruction and Computer-Aided Polyp Detection in Optical Colonoscopy Video Frames , 2016, ArXiv.

[33]  Arie E. Kaufman,et al.  Computer-aided detection of polyps in optical colonoscopy images , 2016, SPIE Medical Imaging.

[34]  Helder Araújo,et al.  A deep learning based fusion of RGB camera information and magnetic localization information for endoscopic capsule robots , 2017, International Journal of Intelligent Robotics and Applications.

[35]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[36]  Helder Araújo,et al.  A fully dense and globally consistent 3D map reconstruction approach for GI tract to enhance therapeutic relevance of the endoscopic capsule robot , 2017, ArXiv.

[37]  Hung-Tat Tsui,et al.  Global Shape from Shading for an Endoscope Image , 1999, MICCAI.

[38]  Mubarak Shah,et al.  Shape from shading using linear approximation , 1994, Image Vis. Comput..

[39]  Helder Araújo,et al.  A non-rigid map fusion-based direct SLAM method for endoscopic capsule robots , 2017, International Journal of Intelligent Robotics and Applications.

[40]  Max Q.-H. Meng,et al.  3D reconstruction of wireless capsule endoscopy images , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[41]  Qingyu Zhao,et al.  The Endoscopogram: A 3D Model Reconstructed from Endoscopic Video Frames , 2016, MICCAI.

[42]  Stephen Lin,et al.  Single-image vignetting correction using radial gradient symmetry , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.