Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots

Abstract Ingestible wireless capsule endoscopy is an emerging minimally invasive diagnostic technology for inspection of the GI tract and diagnosis of a wide range of diseases and pathologies. Medical device companies and many research groups have recently made substantial progresses in converting passive capsule endoscopes to active capsule robots, enabling more accurate, precise, and intuitive detection of the location and size of the diseased areas. Since a reliable real time pose estimation functionality is crucial for actively controlled endoscopic capsule robots, in this study, we propose a monocular visual odometry (VO) method for endoscopic capsule robot operations. Our method lies on the application of the deep recurrent convolutional neural networks (RCNNs) for the visual odometry task, where convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are used for the feature extraction and inference of dynamics across the frames, respectively. Detailed analyses and evaluations made on a real pig stomach dataset proves that our system achieves high translational and rotational accuracies for different types of endoscopic capsule robot trajectories.

[1]  M. Goenka,et al.  Capsule endoscopy: Present status and future expectation. , 2014, World journal of gastroenterology.

[2]  M. O’Donnell,et al.  Sonographic elasticity imaging of acute and chronic deep venous thrombosis in humans. , 2007, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[3]  Metin Sitti,et al.  A 5-D Localization Method for a Magnetically Manipulated Untethered Robot Using a 2-D Array of Hall-Effect Sensors , 2016, IEEE/ASME Transactions on Mechatronics.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Niko Sünderhauf,et al.  On the performance of ConvNet features for place recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Sen Wang,et al.  DeepVO: Towards end-to-end visual odometry with deep Recurrent Convolutional Neural Networks , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[7]  G. Pan,et al.  Swallowable Wireless Capsule Endoscopy: Progress and Technical Challenges , 2011, Gastroenterology research and practice.

[8]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[9]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[10]  Eric Diller,et al.  Biomedical Applications of Untethered Mobile Milli/Microrobots , 2015, Proceedings of the IEEE.

[11]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[13]  Helder Araújo,et al.  Sparse-then-dense alignment-based 3D map reconstruction method for endoscopic capsule robots , 2017, Machine Vision and Applications.

[14]  Yasin Almalioglu,et al.  A Deep Learning Based 6 Degree-of-Freedom Localization Method for Endoscopic Capsule Robots , 2017, ArXiv.

[15]  Weihua Li,et al.  A review of drug delivery systems for capsule endoscopy. , 2014, Advanced drug delivery reviews.

[16]  Jake J. Abbott,et al.  Managing the attractive magnetic force between an untethered magnetically actuated tool and a rotating permanent magnet , 2013, 2013 IEEE International Conference on Robotics and Automation.

[17]  Metin Sitti,et al.  Magnetically actuated soft capsule endoscope for fine-needle aspiration biopsy , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Jake J. Abbott,et al.  An omnidirectional electromagnet for remote manipulation , 2013, 2013 IEEE International Conference on Robotics and Automation.

[19]  Helder Araújo,et al.  A non-rigid map fusion-based direct SLAM method for endoscopic capsule robots , 2017, International Journal of Intelligent Robotics and Applications.

[20]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .

[21]  Metin Sitti,et al.  Biopsy using a Magnetic Capsule Endoscope Carrying, Releasing, and Retrieving Untethered Microgrippers , 2014, IEEE Transactions on Biomedical Engineering.

[22]  Roberto Cipolla,et al.  PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[23]  Mubarak Shah,et al.  Shape from shading using linear approximation , 1994, Image Vis. Comput..

[24]  Helder Araújo,et al.  Six Degree-of-Freedom Localization of Endoscopic Capsule Robots using Recurrent Neural Networks embedded into a Convolutional Neural Network , 2017, ArXiv.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[27]  Daniel Cremers,et al.  Image-Based Localization Using LSTMs for Structured Feature Correlation , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  C. Jia,et al.  Noninvasive ultrasound elasticity imaging (UEI) of Crohn's disease: animal model. , 2008, Ultrasound in medicine & biology.

[29]  Helder Araújo,et al.  A fully dense and globally consistent 3D map reconstruction approach for GI tract to enhance therapeutic relevance of the endoscopic capsule robot , 2017, ArXiv.

[30]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[31]  Michael Talcott,et al.  Magnetically Controllable Gastrointestinal Steering of Video Capsules , 2011, IEEE Transactions on Biomedical Engineering.

[32]  M. Fluckiger,et al.  Ultrasound Emitter Localization in Heterogeneous Media , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[33]  Helder Araujo,et al.  Magnetic-Visual Sensor Fusion based Medical SLAM for Endoscopic Capsule Robot , 2017 .

[34]  Weihua Li,et al.  A Review of Localization Systems for Robotic Endoscopic Capsules , 2012, IEEE Transactions on Biomedical Engineering.

[35]  Metin Sitti,et al.  3-D Localization Method for a Magnetically Actuated Soft Capsule Endoscope and Its Applications , 2013, IEEE Transactions on Robotics.

[36]  Daniel Cremers,et al.  Image-based Localization with Spatial LSTMs , 2016, ArXiv.

[37]  Z. Liao,et al.  Indications and detection, completion, and retention rates of small-bowel capsule endoscopy: a systematic review. , 2010, Gastrointestinal endoscopy.

[38]  Tetsuya Nakamura,et al.  Capsule endoscopy: past, present, and future , 2008, Journal of Gastroenterology.

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  A. Juloski,et al.  Method for navigation and control of a magnetically guided capsule endoscope in the human stomach , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).