Context-Aware Depth and Pose Estimation for Bronchoscopic Navigation

Endobronchial intervention is increasingly used as a minimally invasive means of lung intervention. Vision-based localization approaches are often sensitive to image artifacts in bronchoscopic videos. In this letter, a robust navigation system based on a context-aware depth recovery approach for monocular video images is presented. To handle the artifacts, a conditional generative adversarial learning framework is proposed for reliable depth recovery. The accuracy of depth estimation and camera localization is validated on an in vivo dataset. Both quantitative and qualitative results demonstrate that the depth recovered with the proposed method preserves better structural information of airway lumens in the presence of image artifacts, and the improved camera localization accuracy demonstrates its clinical potential for bronchoscopic navigation.

[1]  Xiangjian He,et al.  Observation-driven adaptive differential evolution and its application to accurate and smooth bronchoscope three-dimensional motion tracking , 2015, Medical Image Anal..

[2]  Alexandru Telea,et al.  An Augmented Fast Marching Method for Computing Skeletons and Centerlines , 2002, VisSym.

[3]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[5]  Takayuki Kitasaka,et al.  Towards hybrid bronchoscope tracking under respiratory motion: evaluation on a dynamic motion phantom , 2010, Medical Imaging.

[6]  Arun Chockalingam,et al.  Transthoracic needle aspiration: the past, present and future. , 2015, Journal of thoracic disease.

[7]  Guang-Zhong Yang,et al.  BRANCH: Bifurcation Recognition for Airway Navigation based on struCtural cHaracteristics , 2017, MICCAI.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  Ralf Eberhardt,et al.  Navigational bronchoscopy in solitary pulmonary nodules , 2017 .

[10]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Debora Gil,et al.  On-Line Lumen Centre Detection in Gastrointestinal and Respiratory Endoscopy , 2013, CLIP.

[12]  Kensaku Mori,et al.  A Discriminative Structural Similarity Measure and its Application to Video-Volume Registration for Endoscope Three-Dimensional Motion Tracking , 2014, IEEE Transactions on Medical Imaging.

[13]  F. Lindseth,et al.  Navigated Bronchoscopy: A Technical Review , 2014, Journal of bronchology & interventional pulmonology.

[14]  Noah Snavely,et al.  Unsupervised Learning of Depth and Ego-Motion from Video , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Kensaku Mori,et al.  Advanced Endoscopic Navigation: Surgical Big Data, Methodology, and Applications. , 2018, Annual review of biomedical engineering.

[16]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[17]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[18]  Guang-Zhong Yang,et al.  Robust camera localisation with depth reconstruction for bronchoscopic navigation , 2015, International Journal of Computer Assisted Radiology and Surgery.

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[21]  Ian D. Reid,et al.  Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Faisal Mahmood,et al.  Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training , 2017, IEEE Transactions on Medical Imaging.

[23]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[24]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[25]  Guang-Zhong Yang,et al.  Metric depth recovery from monocular images using Shape-from-Shading and specularities , 2012, 2012 19th IEEE International Conference on Image Processing.

[26]  Toshimitsu Kaneko,et al.  Deep monocular 3D reconstruction for assisted navigation in bronchoscopy , 2017, International Journal of Computer Assisted Radiology and Surgery.

[27]  Guang-Zhong Yang,et al.  Patient-specific bronchoscope simulation with pq-space-based 2D/3D registration , 2004, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[28]  Herman Alexander Jaeger,et al.  Automated Catheter Navigation With Electromagnetic Image Guidance , 2017, IEEE Transactions on Biomedical Engineering.

[29]  Guosheng Lin,et al.  Deep convolutional neural fields for depth estimation from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Eric J. Seibel,et al.  In Vivo Validation of a Hybrid Tracking System for Navigation of an Ultrathin Bronchoscope Within Peripheral Airways , 2010, IEEE Transactions on Biomedical Engineering.