Endo-VMFuseNet: Deep Visual-Magnetic Sensor Fusion Approach for Uncalibrated, Unsynchronized and Asymmetric Endoscopic Capsule Robot Localization Data

In the last decade, researchers and medical device companies have made major advances towards transforming passive capsule endoscopes into active medical robots. One of the major challenges is to endow capsule robots with accurate perception of the environment inside the human body, which will provide necessary information and enable improved medical procedures. We extend the success of deep learning approaches from various research fields to the problem of uncalibrated, asynchronous, and asymmetric sensor fusion for endoscopic capsule robots. The results performed on real pig stomach datasets show that our method achieves sub-millimeter precision for both translational and rotational movements and contains various advantages over traditional sensor fusion techniques.

[1]  D. Fischer,et al.  Capsule endoscopy: the localization system. , 2004, Gastrointestinal endoscopy clinics of North America.

[2]  Kaveh Pahlavan,et al.  Performance bounds for RF positioning of endoscopy camera capsules , 2011, 2011 IEEE Topical Conference on Biomedical Wireless Technologies, Networks, and Sensing Systems.

[3]  Baris Fidan,et al.  Adaptive Wireless Biomedical Capsule Tracking Based on Magnetic Sensing , 2017, Int. J. Wirel. Inf. Networks.

[4]  Tim C. Lueth,et al.  Navigation of a robotic capsule endoscope with a novel ultrasound tracking system , 2013 .

[5]  Metin Sitti,et al.  Magnetically actuated soft capsule endoscope for fine-needle aspiration biopsy , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[6]  Helder Araújo,et al.  Deep EndoVO: A recurrent convolutional neural network (RCNN) based visual odometry approach for endoscopic capsule robots , 2017, Neurocomputing.

[7]  K. Arshak,et al.  Capsule tracking in the GI tract: a novel microcontroller based solution , 2006, Proceedings of the 2006 IEEE Sensors Applications Symposium, 2006..

[8]  Roberto Cipolla,et al.  PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[9]  Pietro Valdastri,et al.  Jacobian-Based Iterative Method for Magnetic Localization in Robotic Capsule Endoscopy , 2016, IEEE Transactions on Robotics.

[10]  Baris Fidan,et al.  Localization and Tracking of Implantable Biomedical Sensors , 2017, Sensors.

[11]  Matthias Nießner,et al.  BundleFusion , 2016, TOGS.

[12]  Helder Araújo,et al.  A fully dense and globally consistent 3D map reconstruction approach for GI tract to enhance therapeutic relevance of the endoscopic capsule robot , 2017, ArXiv.

[13]  Stefan Leutenegger,et al.  ElasticFusion: Dense SLAM Without A Pose Graph , 2015, Robotics: Science and Systems.

[14]  Sen Wang,et al.  VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem , 2017, AAAI.

[15]  Guang-Zhong Yang,et al.  Metric depth recovery from monocular images using Shape-from-Shading and specularities , 2012, 2012 19th IEEE International Conference on Image Processing.

[16]  Max Q.-H. Meng,et al.  A novel radio propagation radiation model for location of the capsule in GI tract , 2009, 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[17]  Metin Sitti,et al.  3-D Localization Method for a Magnetically Actuated Soft Capsule Endoscope and Its Applications , 2013, IEEE Transactions on Robotics.

[18]  Guang-Zhong Yang,et al.  Intra-operative monocular 3D reconstruction for image-guided navigation in active locomotion capsule endoscopy , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[19]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[20]  Helder Araújo,et al.  Six Degree-of-Freedom Localization of Endoscopic Capsule Robots using Recurrent Neural Networks embedded into a Convolutional Neural Network , 2017, ArXiv.

[21]  Jake J. Abbott,et al.  Localization method for a magnetic capsule endoscope propelled by a rotating magnetic dipole field , 2013, 2013 IEEE International Conference on Robotics and Automation.

[22]  Helder Araújo,et al.  Sparse-then-dense alignment-based 3D map reconstruction method for endoscopic capsule robots , 2017, Machine Vision and Applications.

[23]  Kaveh Pahlavan,et al.  Design, Implementation, and Fundamental Limits of Image and RF Based Wireless Capsule Endoscopy Hybrid Localization , 2016, IEEE Transactions on Mobile Computing.

[24]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[25]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[26]  Helder Araújo,et al.  A Non-Rigid Map Fusion-Based RGB-Depth SLAM Method for Endoscopic Capsule Robots , 2017, ArXiv.

[27]  Kaveh Pahlavan,et al.  Hybrid Localization of Microrobotic Endoscopic Capsule Inside Small Intestine by Data Fusion of Vision and RF Sensors , 2015, IEEE Sensors Journal.

[28]  Weihua Li,et al.  An Effective Localization Method for Robotic Endoscopic Capsules Using Multiple Positron Emission Markers , 2014, IEEE Transactions on Robotics.

[29]  Metin Sitti,et al.  A 5-D Localization Method for a Magnetically Manipulated Untethered Robot Using a 2-D Array of Hall-Effect Sensors , 2016, IEEE/ASME Transactions on Mechatronics.

[30]  Yunxing Ye,et al.  Bounds on RF cooperative localization for video capsule endoscopy , 2013 .

[31]  Helder Araujo,et al.  Magnetic-Visual Sensor Fusion based Medical SLAM for Endoscopic Capsule Robot , 2017 .

[32]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[33]  Helder Araújo,et al.  EndoSensorFusion: Particle Filtering-Based Multi-Sensory Data Fusion with Switching State-Space Model for Endoscopic Capsule Robots , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Yongxin Zhu,et al.  Design and Implementation of a High Resolution Localization System for In-Vivo Capsule Endoscopy , 2009, 2009 Eighth IEEE International Conference on Dependable, Autonomic and Secure Computing.

[35]  Baris Fidan,et al.  Adaptive magnetic sensing based wireless capsule localization , 2016, 2016 10th International Symposium on Medical Information and Communication Technology (ISMICT).