SynthMorph: Learning Contrast-Invariant Registration Without Acquired Images

We introduce a learning strategy for contrast-invariant image registration without requiring imaging data. While classical registration methods accurately estimate the spatial correspondence between images, they solve a costly optimization problem for every image pair. Learning-based techniques are fast at test time, but can only register images with image contrast and geometric content that are similar to those available during training. We focus on removing this image-data dependency of learning methods. Our approach leverages a generative model for diverse label maps and images that exposes networks to a wide range of variability during training, forcing them to learn features invariant to image type (contrast). This strategy results in powerful networks trained to generalize to a broad array of real input images. We present extensive experiments, with a focus on 3D neuroimaging, showing that this strategy enables robust registration of arbitrary image contrasts without the need to retrain for new modalities. We demonstrate registration accuracy that most often surpasses the state of the art both within and across modalities, using a single model. Critically, we show that input labels from which we synthesize images need not be of actual anatomy: training on randomly generated geometric shapes also results in competitive registration performance, albeit slightly less accurate, while alleviating the dependency on real data of any kind. Our code is available at: this http URL

[1]  Max A. Viergever,et al.  A deep learning framework for unsupervised affine and deformable image registration , 2018, Medical Image Anal..

[2]  Snehashis Roy,et al.  Cross contrast multi‐channel image registration using image synthesis for MR brain images , 2017, Medical Image Anal..

[3]  Tom Vercauteren,et al.  Diffeomorphic demons: Efficient non-parametric image registration , 2009, NeuroImage.

[4]  Snehashis Roy,et al.  MR to CT registration of brains using image synthesis , 2014, Medical Imaging.

[5]  L. R. Dice Measures of the Amount of Ecologic Association Between Species , 1945 .

[6]  Alain Trouvé,et al.  Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms , 2005, International Journal of Computer Vision.

[7]  Dinggang Shen,et al.  Scalable High-Performance Image Registration Framework by Unsupervised Deep Feature Representations Learning , 2016, IEEE Transactions on Biomedical Engineering.

[8]  Daniel Rueckert,et al.  Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations , 2019, IPMI.

[9]  Frank Sauer,et al.  Learning Based Non-rigid Multi-modal Image Registration Using Kullback-Leibler Divergence , 2005, MICCAI.

[10]  Justin P. Haldar,et al.  Co-registration and distortion correction of diffusion and anatomical images based on inverse contrast normalization , 2015, NeuroImage.

[11]  Yong Fan,et al.  Non-rigid image registration using fully convolutional networks with deep self-supervision , 2017, ArXiv.

[12]  Hervé Delingette,et al.  Learning a Probabilistic Model for Diffeomorphic Registration , 2018, IEEE Transactions on Medical Imaging.

[13]  Jan Rühaak,et al.  A fully parallel algorithm for multimodal image registration using normalized gradient fields , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[14]  Polina Golland,et al.  Quantification and Analysis of Large Multimodal Clinical Image Studies: Application to Stroke , 2013, MBIA.

[15]  Ken Perlin,et al.  [Computer Graphics]: Three-Dimensional Graphics and Realism , 2022 .

[16]  Essa Yacoub,et al.  The Lifespan Human Connectome Project in Aging: An overview , 2019, NeuroImage.

[17]  Josien P. W. Pluim,et al.  Pulmonary CT Registration Through Supervised Learning With Convolutional Neural Networks , 2019, IEEE Transactions on Medical Imaging.

[18]  Ben Glocker,et al.  Is Synthesizing MRI Contrast Useful for Inter-modality Analysis? , 2013, MICCAI.

[19]  A. Dale,et al.  Whole Brain Segmentation Automated Labeling of Neuroanatomical Structures in the Human Brain , 2002, Neuron.

[20]  Hugues Benoit-Cattin,et al.  Intensity non-uniformity correction in MRI: Existing methods and their validation , 2006, Medical Image Anal..

[21]  Martin J. Graves,et al.  MRI from Picture to Proton , 2017 .

[22]  C. Jack,et al.  Alzheimer's Disease Neuroimaging Initiative , 2008 .

[23]  Sébastien Ourselin,et al.  Fast free-form deformation using graphics processing units , 2010, Comput. Methods Programs Biomed..

[24]  Nicholas Ayache,et al.  A Log-Euclidean Framework for Statistics on Diffeomorphisms , 2006, MICCAI.

[25]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[26]  Maxime Sermesant,et al.  SVF-Net: Learning Deformable Image Registration Using Shape Matching , 2017, MICCAI.

[27]  Thomas E. Nichols,et al.  Extending the Human Connectome Project across ages: Imaging protocols for the Lifespan Development and Aging projects , 2018, NeuroImage.

[28]  Michael Brady,et al.  MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration , 2012, Medical Image Anal..

[29]  Xiao Yang,et al.  Fast Predictive Image Registration , 2016, LABELS/DLMIA@MICCAI.

[30]  Hervé Delingette,et al.  Robust Non-rigid Registration Through Agent-Based Action Learning , 2017, MICCAI.

[31]  Ender Konukoglu,et al.  Semi-Supervised and Task-Driven Data Augmentation , 2019, IPMI.

[32]  J. Mugler,et al.  Three‐dimensional magnetization‐prepared rapid gradient‐echo imaging (3D MP RAGE) , 1990, Magnetic resonance in medicine.

[33]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[34]  Nicholas Ayache,et al.  The Correlation Ratio as a New Similarity Measure for Multimodal Image Registration , 1998, MICCAI.

[35]  Ben Glocker,et al.  Supervoxel Classification Forests for Estimating Pairwise Image Correspondences , 2015, MLMI.

[36]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[37]  Sébastien Ourselin,et al.  Evaluation of Six Registration Methods for the Human Abdomen on Clinically Acquired CT , 2016, IEEE Transactions on Biomedical Engineering.

[38]  Diego H. Milone,et al.  Learning Deformable Registration of Medical Images with Anatomical Constraints , 2020, Neural Networks.

[39]  Bruce Fischl,et al.  Highly accurate inverse consistent registration: A robust approach , 2010, NeuroImage.

[40]  Nassir Navab,et al.  Entropy and Laplacian images: Structural representations for multi-modal registration , 2012, Medical Image Anal..

[41]  Mert R. Sabuncu,et al.  VoxelMorph: A Learning Framework for Deformable Medical Image Registration , 2018, IEEE Transactions on Medical Imaging.

[42]  Jan Rühaak,et al.  A fast and accurate parallel algorithm for non-linear image registration using Normalized Gradient fields , 2014, ISBI.

[43]  Stefan Heldmann,et al.  Enhancing Label-Driven Deep Deformable Image Registration with Local Distance Metrics for State-of-the-Art Cardiac Motion Tracking , 2018, Bildverarbeitung für die Medizin.

[44]  Alexander Derksen,et al.  Parallel and memory efficient multimodal image registration for radiotherapy using normalized gradient fields , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[45]  Eldad Haber,et al.  Intensity Gradient Based Registration and Fusion of Multi-modal Images , 2006, MICCAI.

[46]  Stephen M. Smith,et al.  Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm , 2001, IEEE Transactions on Medical Imaging.

[47]  Marc Niethammer,et al.  Quicksilver: Fast predictive image registration – A deep learning approach , 2017, NeuroImage.

[48]  P. Ellen Grant,et al.  Fetal Pose Estimation in Volumetric MRI using a 3D Convolution Neural Network , 2019, MICCAI.

[49]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[50]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[51]  John Ashburner,et al.  A fast diffeomorphic image registration algorithm , 2007, NeuroImage.

[52]  Michael Brady,et al.  Deformable image registration by combining uncertainty estimates from supervoxel belief propagation , 2016, Medical Image Anal..

[53]  Mert R. Sabuncu,et al.  Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces , 2019, Medical Image Anal..

[54]  Ender Konukoglu,et al.  Generative Adversarial Networks for MR-CT Deformable Image Registration , 2018, ArXiv.

[55]  Zujun Hou,et al.  A Review on MR Image Intensity Inhomogeneity Correction , 2006, Int. J. Biomed. Imaging.

[56]  W. Eric L. Grimson,et al.  Adaptive Segmentation of MRI Data , 1995, CVRMed.

[57]  Brian B. Avants,et al.  Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain , 2008, Medical Image Anal..

[58]  Nicholas Ayache,et al.  LCC-Demons: A robust and accurate symmetric diffeomorphic registration algorithm , 2013, NeuroImage.

[59]  Konstantinos Kamnitsas,et al.  Unsupervised domain adaptation in brain lesion segmentation with adversarial networks , 2016, IPMI.

[60]  Boudewijn P. F. Lelieveldt,et al.  Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks , 2017, MICCAI.

[61]  Jürgen Weese,et al.  Landmark-based elastic registration using approximating thin-plate splines , 2001, IEEE Transactions on Medical Imaging.

[62]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[63]  Michael Brady,et al.  Phase mutual information as a similarity measure for registration , 2005, Medical Image Anal..

[64]  Marc Modat,et al.  Label-driven weakly-supervised learning for multimodal deformarle image registration , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[65]  Bruce Fischl,et al.  A Learning Strategy for Contrast-agnostic MRI Segmentation , 2020, MIDL.

[66]  Karl J. Friston,et al.  Human Brain Function , 1997 .

[67]  R R Edelman,et al.  Contrast in rapid MR imaging: T1- and T2-weighted imaging. , 1987, Journal of computer assisted tomography.

[68]  Frédo Durand,et al.  Data augmentation using learned transforms for one-shot medical image segmentation , 2019, ArXiv.

[69]  Koenraad Van Leemput,et al.  Fast and sequence-adaptive whole-brain segmentation using parametric Bayesian modeling , 2016, NeuroImage.

[70]  Sébastien Ourselin,et al.  Weakly-supervised convolutional neural networks for multimodal image registration , 2018, Medical Image Anal..

[71]  Arno Klein,et al.  Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration , 2009, NeuroImage.

[72]  Koenraad Van Leemput,et al.  A unifying framework for partial volume segmentation of brain MR images , 2003, IEEE Transactions on Medical Imaging.