A deep learning based fusion of RGB camera information and magnetic localization information for endoscopic capsule robots

A reliable, real time localization functionality is crutial for actively controlled capsule endoscopy robots, which are an emerging, minimally invasive diagnostic and therapeutic technology for the gastrointestinal (GI) tract. In this study, we extend the success of deep learning approaches from various research fields to the problem of sensor fusion for endoscopic capsule robots. We propose a multi-sensor fusion based localization approach which combines endoscopic camera information and magnetic sensor based localization information. The results performed on real pig stomach dataset show that our method achieves sub-millimeter precision for both translational and rotational movements.

[1]  Baris Fidan,et al.  Adaptive Wireless Biomedical Capsule Tracking Based on Magnetic Sensing , 2017, Int. J. Wirel. Inf. Networks.

[2]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[3]  KanadeTakeo,et al.  Adapting optical-flow to measure object motion in reflectance and x-ray image sequences (abstract only) , 1984 .

[4]  K. Arshak,et al.  Capsule tracking in the GI tract: a novel microcontroller based solution , 2006, Proceedings of the 2006 IEEE Sensors Applications Symposium, 2006..

[5]  Jake J. Abbott,et al.  Localization method for a magnetic capsule endoscope propelled by a rotating magnetic dipole field , 2013, 2013 IEEE International Conference on Robotics and Automation.

[6]  Pietro Valdastri,et al.  Jacobian-Based Iterative Method for Magnetic Localization in Robotic Capsule Endoscopy , 2016, IEEE Transactions on Robotics.

[7]  Baris Fidan,et al.  Localization and Tracking of Implantable Biomedical Sensors , 2017, Sensors.

[8]  Kaveh Pahlavan,et al.  Design, Implementation, and Fundamental Limits of Image and RF Based Wireless Capsule Endoscopy Hybrid Localization , 2016, IEEE Transactions on Mobile Computing.

[9]  Metin Sitti,et al.  A 5-D Localization Method for a Magnetically Manipulated Untethered Robot Using a 2-D Array of Hall-Effect Sensors , 2016, IEEE/ASME Transactions on Mechatronics.

[10]  D. Fischer,et al.  Capsule endoscopy: the localization system. , 2004, Gastrointestinal endoscopy clinics of North America.

[11]  Takeo Kanade,et al.  Adapting optical-flow to measure object motion in reflectance and x-ray image sequences (abstract only) , 1984, COMG.

[12]  Metin Sitti,et al.  3-D Localization Method for a Magnetically Actuated Soft Capsule Endoscope and Its Applications , 2013, IEEE Transactions on Robotics.

[13]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[14]  Kaveh Pahlavan,et al.  Hybrid Localization of Microrobotic Endoscopic Capsule Inside Small Intestine by Data Fusion of Vision and RF Sensors , 2015, IEEE Sensors Journal.

[15]  Yunxing Ye,et al.  Bounds on RF cooperative localization for video capsule endoscopy , 2013 .

[16]  Helder Araújo,et al.  A non-rigid map fusion-based direct SLAM method for endoscopic capsule robots , 2017, International Journal of Intelligent Robotics and Applications.

[17]  Max Q.-H. Meng,et al.  A novel radio propagation radiation model for location of the capsule in GI tract , 2009, 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[18]  Yongxin Zhu,et al.  Design and Implementation of a High Resolution Localization System for In-Vivo Capsule Endoscopy , 2009, 2009 Eighth IEEE International Conference on Dependable, Autonomic and Secure Computing.

[19]  Baris Fidan,et al.  Adaptive magnetic sensing based wireless capsule localization , 2016, 2016 10th International Symposium on Medical Information and Communication Technology (ISMICT).

[20]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[21]  Weihua Li,et al.  An Effective Localization Method for Robotic Endoscopic Capsules Using Multiple Positron Emission Markers , 2014, IEEE Transactions on Robotics.

[22]  Sen Wang,et al.  VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem , 2017, AAAI.

[23]  Kaveh Pahlavan,et al.  Performance bounds for RF positioning of endoscopy camera capsules , 2011, 2011 IEEE Topical Conference on Biomedical Wireless Technologies, Networks, and Sensing Systems.

[24]  Tim C. Lueth,et al.  Navigation of a robotic capsule endoscope with a novel ultrasound tracking system , 2013 .

[25]  Metin Sitti,et al.  Magnetically actuated soft capsule endoscope for fine-needle aspiration biopsy , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).