Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots

In the last decade, many medical companies and research groups have tried to convert passive capsule endoscopes as an emerging and minimally invasive diagnostic technology into actively steerable endoscopic capsule robots which will provide more intuitive disease detection, targeted drug delivery and biopsy-like operations in the gastrointestinal(GI) tract. In this study, we introduce a fully unsupervised, realtime odometry and depth learner for monocular endoscopic capsule robots. We establish the supervision by warping view sequences and assigning the re-projection minimization to the loss function, which we adopt in multi-view pose estimation and single-view depth estimation network. Detailed quantitative and qualitative analyses of the proposed framework performed on non-rigidly deformable ex-vivo porcine stomach datasets proves the effectiveness of the method in terms of motion estimation and depth recovery.

[1]  Richard Szeliski,et al.  Prediction error as a quality metric for motion and stereo , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[3]  Stanislav Emelianov,et al.  Sonographic Elasticity Imaging of Acute and Chronic Deep Venous Thrombosis in Humans , 2006 .

[4]  M. Fluckiger,et al.  Ultrasound Emitter Localization in Heterogeneous Media , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[5]  Tetsuya Nakamura,et al.  Capsule endoscopy: past, present, and future , 2008, Journal of Gastroenterology.

[6]  C. Jia,et al.  Noninvasive ultrasound elasticity imaging (UEI) of Crohn's disease: animal model. , 2008, Ultrasound in medicine & biology.

[7]  Michael Talcott,et al.  Magnetically Controllable Gastrointestinal Steering of Video Capsules , 2011, IEEE Transactions on Biomedical Engineering.

[8]  A. Juloski,et al.  Method for navigation and control of a magnetically guided capsule endoscope in the human stomach , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[9]  Weihua Li,et al.  A Review of Localization Systems for Robotic Endoscopic Capsules , 2012, IEEE Transactions on Biomedical Engineering.

[10]  Jake J. Abbott,et al.  An omnidirectional electromagnet for remote manipulation , 2013, 2013 IEEE International Conference on Robotics and Automation.

[11]  Jake J. Abbott,et al.  Managing the attractive magnetic force between an untethered magnetically actuated tool and a rotating permanent magnet , 2013, 2013 IEEE International Conference on Robotics and Automation.

[12]  Metin Sitti,et al.  3-D Localization Method for a Magnetically Actuated Soft Capsule Endoscope and Its Applications , 2013, IEEE Transactions on Robotics.

[13]  Weihua Li,et al.  A review of drug delivery systems for capsule endoscopy. , 2014, Advanced drug delivery reviews.

[14]  M. Goenka,et al.  Capsule endoscopy: Present status and future expectation. , 2014, World journal of gastroenterology.

[15]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[16]  Metin Sitti,et al.  Biopsy using a Magnetic Capsule Endoscope Carrying, Releasing, and Retrieving Untethered Microgrippers , 2014, IEEE Transactions on Biomedical Engineering.

[17]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[18]  Eric Diller,et al.  Biomedical Applications of Untethered Mobile Milli/Microrobots , 2015, Proceedings of the IEEE.

[19]  Thomas Brox,et al.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Jitendra Malik,et al.  View Synthesis by Appearance Flow , 2016, ECCV.

[21]  John Flynn,et al.  Deep Stereo: Learning to Predict New Views from the World's Imagery , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Helder Araújo,et al.  A fully dense and globally consistent 3D map reconstruction approach for GI tract to enhance therapeutic relevance of the endoscopic capsule robot , 2017, ArXiv.

[23]  Yasin Almalioglu,et al.  A Deep Learning Based 6 Degree-of-Freedom Localization Method for Endoscopic Capsule Robots , 2017, ArXiv.

[24]  Helder Araújo,et al.  A non-rigid map fusion-based direct SLAM method for endoscopic capsule robots , 2017, International Journal of Intelligent Robotics and Applications.

[25]  Noah Snavely,et al.  Unsupervised Learning of Depth and Ego-Motion from Video , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Helder Araújo,et al.  A Non-Rigid Map Fusion-Based RGB-Depth SLAM Method for Endoscopic Capsule Robots , 2017, ArXiv.

[27]  Helder Araújo,et al.  Sparse-then-dense alignment-based 3D map reconstruction method for endoscopic capsule robots , 2017, Machine Vision and Applications.

[28]  Helder Araújo,et al.  Six Degree-of-Freedom Localization of Endoscopic Capsule Robots using Recurrent Neural Networks embedded into a Convolutional Neural Network , 2017, ArXiv.

[29]  Cordelia Schmid,et al.  SfM-Net: Learning of Structure and Motion from Video , 2017, ArXiv.

[30]  Yasin Almalioglu,et al.  Endo-VMFuseNet: Deep Visual-Magnetic Sensor Fusion Approach for Uncalibrated, Unsynchronized and Asymmetric Endoscopic Capsule Robot Localization Data , 2017, ArXiv.

[31]  Helder Araújo,et al.  EndoSensorFusion: Particle Filtering-Based Multi-Sensory Data Fusion with Switching State-Space Model for Endoscopic Capsule Robots , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[32]  Helder Araújo,et al.  Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots , 2017, Neurocomputing.