Guiding multimodal registration with learned optimization updates

HighlightsSupervised learning method to learn the transformation aligning a pair of images directly from their appearance.Application to a broad range of imaging modalities.Registration with increased capture range when compared to other similarity metrics.Flexibility to be used with different parametrizations. Graphical abstract No Caption available. Abstract In this paper, we address the multimodal registration problem from a novel perspective, aiming to predict the transformation aligning images directly from their visual appearance. We formulate the prediction as a supervised regression task, with joint image descriptors as input and the output are the parameters of the transformation that guide the moving image towards alignment. We model the joint local appearance with context aware descriptors that capture both local and global cues simultaneously in the two modalities, while the regression function is based on the gradient boosted trees method capable of handling the very large contextual feature space. The good properties of our predictions allow us to couple them with a simple gradient‐based optimization for the final registration. Our approach can be applied to any transformation parametrization as well as a broad range of modality pairs. Our method learns the relationship between the intensity distributions of a pair of modalities by using prior knowledge in the form of a small training set of aligned image pairs (in the order of 1–5 in our experiments). We demonstrate the flexibility and generality of our method by evaluating its performance on a variety of multimodal imaging pairs obtained from two publicly available datasets, RIRE (brain MR, CT and PET) and IXI (brain MR). We also show results for the very challenging deformable registration of Intravascular Ultrasound and Histology images. In these experiments, our approach has a larger capture range when compared to other state‐of‐the‐art methods, while improving registration accuracy in complex cases.

[1]  Stephen M. Pizer,et al.  2D/3D image registration using regression learning , 2013, Comput. Vis. Image Underst..

[2]  Nassir Navab,et al.  Entropy and Laplacian images: Structural representations for multi-modal registration , 2012, Medical Image Anal..

[3]  Antonio Criminisi,et al.  Decision Forests with Long-Range Spatial Context for Organ Localization in CT Volumes , 2009 .

[4]  Rich Caruana,et al.  An empirical comparison of supervised learning algorithms , 2006, ICML.

[5]  Amin Katouzian,et al.  Challenges in tissue characterization from backscattered intravascular ultrasound signals , 2007, SPIE Medical Imaging.

[6]  Xiao Yang,et al.  Fast Predictive Image Registration , 2016, LABELS/DLMIA@MICCAI.

[7]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[8]  Nassir Navab,et al.  Learning Optimization Updates for Multimodal Registration , 2016, MICCAI.

[9]  Mert R. Sabuncu,et al.  Using Spanning Graphs for Efficient Image Registration , 2008, IEEE Transactions on Image Processing.

[10]  Fernando De la Torre,et al.  Supervised Descent Method and Its Applications to Face Alignment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Gerald Q. Maguire,et al.  Comparison and evaluation of retrospective intermodality brain image registration techniques. , 1997, Journal of computer assisted tomography.

[12]  Max A. Viergever,et al.  f-information measures in medical image registration , 2004, IEEE Transactions on Medical Imaging.

[13]  Nikos Paragios,et al.  Deformable Medical Image Registration: A Survey , 2013, IEEE Transactions on Medical Imaging.

[14]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[15]  Pierrick Coupé,et al.  3D Rigid Registration of Intraoperative Ultrasound and Preoperative MR Brain Images Based on Hyperechogenic Structures , 2012, Int. J. Biomed. Imaging.

[16]  Daniel Rueckert,et al.  Structured Decision Forests for Multi-modal Ultrasound Image Registration , 2015, MICCAI.

[17]  Jian Sun,et al.  Face Alignment by Explicit Shape Regression , 2012, International Journal of Computer Vision.

[18]  Michael Brady,et al.  Towards Realtime Multimodal Fusion for Image-Guided Interventions Using Self-similarities , 2013, MICCAI.

[19]  Nassir Navab,et al.  IVUS-Histology Image Registration , 2012, WBIR.

[20]  Shaohua Kevin Zhou,et al.  Cross-Domain Synthesis of Medical Images Using Efficient Location-Sensitive Deep Network , 2015, MICCAI.

[21]  Caroline Petitjean,et al.  A review of 3 D / 2 D registration methods for image-guided interventions , 2016 .

[22]  Li Zhang,et al.  Deep similarity learning for multimodal medical images , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[23]  Guy Marchal,et al.  Multimodality image registration by maximization of mutual information , 1997, IEEE Transactions on Medical Imaging.

[24]  Pietro Perona,et al.  Cascaded pose regression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[26]  William M. Wells,et al.  Multi-modal Image Registration Using Dirichlet-Encoded Prior Information , 2006, WBIR.

[27]  Michel Dhome,et al.  Hyperplane Approximation for Template Matching , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Yaozong Gao,et al.  Learning‐based deformable image registration for infant MR images in the first year of life , 2017, Medical physics.

[29]  Michael Brady,et al.  MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration , 2012, Medical Image Anal..

[30]  David R. Haynor,et al.  Nonrigid multimodality image registration , 2001, SPIE Medical Imaging.

[31]  Torsten Rohlfing,et al.  Image Similarity and Tissue Overlaps as Surrogates for Image Registration Accuracy: Widely Used but Unreliable , 2012, IEEE Transactions on Medical Imaging.

[32]  Nassir Navab,et al.  Scale-Adaptive Forest Training via an Efficient Feature Sampling Scheme , 2015, MICCAI.

[33]  Dinggang Shen,et al.  A General Fast Registration Framework by Learning Deformation–Appearance Correlation , 2012, IEEE Transactions on Image Processing.

[34]  G. Marchal,et al.  Multi-modal volume registration by maximization of mutual information , 1997 .

[35]  Max A. Viergever,et al.  Mutual-information-based registration of medical images: a survey , 2003, IEEE Transactions on Medical Imaging.

[36]  Mandy Eberhart,et al.  Decision Forests For Computer Vision And Medical Image Analysis , 2016 .

[37]  Nikos Paragios,et al.  Boosted metric learning for 3D multi-modal deformable registration , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[38]  Nassir Navab,et al.  Personalized, relevance-based Multimodal Robotic Imaging and augmented reality for Computer Assisted Interventions , 2016, Medical Image Anal..

[39]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[40]  Bernhard Schölkopf,et al.  Learning similarity measure for multi-modal 3D image registration , 2009, CVPR 2009.

[41]  Arthur W. Toga,et al.  Learning based coarse-to-fine image registration , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Dorin Comaniciu,et al.  An Artificial Agent for Anatomical Landmark Detection in Medical Images , 2016, MICCAI.

[43]  Nassir Navab,et al.  Simulation and Fully Automatic Multimodal Registration of Medical Ultrasound , 2007, MICCAI.

[44]  Nikos Komodakis,et al.  A Deep Metric for Multimodal Registration , 2016, MICCAI.