Dilated FCN for Multi-Agent 2D/3D Medical Image Registration

2D/3D image registration to align a 3D volume and 2D X-ray images is a challenging problem due to its ill-posed nature and various artifacts presented in 2D X-ray images. In this paper, we propose a multi-agent system with an auto attention mechanism for robust and efficient 2D/3D image registration. Specifically, an individual agent is trained with dilated Fully Convolutional Network (FCN) to perform registration in a Markov Decision Process (MDP) by observing a local region, and the final action is then taken based on the proposals from multiple agents and weighted by their corresponding confidence levels. The contributions of this paper are threefold. First, we formulate 2D/3D registration as a MDP with observations, actions, and rewards properly defined with respect to X-ray imaging systems. Second, to handle various artifacts in 2D X-ray images, multiple local agents are employed efficiently via FCN-based structures, and an auto attention mechanism is proposed to favor the proposals from regions with more reliable visual cues. Third, a dilated FCN-based training mechanism is proposed to significantly reduce the Degree of Freedom in the simulation of registration environment, and drastically improve training efficiency by an order of magnitude compared to standard CNN-based training method. We demonstrate that the proposed method achieves high robustness on both spine cone beam Computed Tomography data with a low signal-to-noise ratio and data from minimally invasive spine surgery where severe image artifacts and occlusions are presented due to metal screws and guide wires, outperforming other state-of-the-art methods (single agent-based and optimization-based) by a large margin.

[1]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[2]  Jérôme Schmid,et al.  Segmentation of X-ray Images by 3D-2D Registration Based on Multibody Physics , 2014, ACCV.

[3]  Dinggang Shen,et al.  Learning-based deformable registration of MR brain images , 2006, IEEE Transactions on Medical Imaging.

[4]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[6]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[8]  Vincent Lepetit,et al.  Learning descriptors for object recognition and 3D pose estimation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Stefan Klein,et al.  Registration-by-regression of coronary CTA and X-ray angiography , 2017, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[11]  Albert C. S. Chung,et al.  A Global Optimization Strategy for 3D-2D Registration of Vascular Images , 2006, BMVC.

[12]  Guy Marchal,et al.  Multimodality image registration by maximization of mutual information , 1997, IEEE Transactions on Medical Imaging.

[13]  A Uneri,et al.  Intraoperative evaluation of device placement in spine surgery using known-component 3D–2D image registration , 2017, Physics in medicine and biology.

[14]  Rüdiger Westermann,et al.  Acceleration techniques for GPU-based volume rendering , 2003, IEEE Visualization, 2003. VIS 2003..

[15]  Dorin Comaniciu,et al.  An Artificial Agent for Robust Image Registration , 2016, AAAI.

[16]  Stephen M. Pizer,et al.  2D/3D image registration using regression learning , 2013, Comput. Vis. Image Underst..

[17]  Gabor Fichtinger,et al.  Monitoring tumor motion by real time 2D/3D registration during radiotherapy , 2012, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[18]  Rui Liao,et al.  Learning CNNs with Pairwise Domain Adaption for Real-Time 6DoF Ultrasound Transducer Detection and Tracking from X-Ray Images , 2017, MICCAI.

[19]  Caroline Petitjean,et al.  A review of 3 D / 2 D registration methods for image-guided interventions , 2016 .

[20]  Leo Joskowicz,et al.  Effective Intensity-Based 2D/3D Rigid Registration between Fluoroscopic X-Ray and CT , 2003, MICCAI.

[21]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[22]  Peter Rossmanith,et al.  Simulated Annealing , 2008, Taschenbuch der Algorithmen.

[23]  Z. Jane Wang,et al.  A CNN Regression Approach for Real-Time 2D/3D Registration , 2016, IEEE Transactions on Medical Imaging.

[24]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Rui Liao,et al.  A hybrid method for 2-D/3-D registration between 3-D volumes and 2-D angiography for trans-catheter aortic valve implantation (TAVI) , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[27]  A Uneri,et al.  3D–2D image registration for target localization in spine surgery: investigation of similarity metrics providing robustness to content mismatch , 2016, Physics in medicine and biology.

[28]  Emile H. L. Aarts,et al.  Simulated Annealing: Theory and Applications , 1987, Mathematics and Its Applications.

[29]  Christin Wirth The Essential Physics of Medical Imaging , 2003, European Journal of Nuclear Medicine and Molecular Imaging.

[30]  Jochen Trumpf,et al.  L1 rotation averaging using the Weiszfeld algorithm , 2011, CVPR 2011.