Medical Image Registration Using Deep Neural Networks: A Comprehensive Review

Image-guided interventions are saving the lives of a large number of patients where the image registration problem should indeed be considered as the most complex and complicated issue to be tackled. On the other hand, the recently huge progress in the field of machine learning made by the possibility of implementing deep neural networks on the contemporary many-core GPUs opened up a promising window to challenge with many medical applications, where the registration is not an exception. In this paper, a comprehensive review on the state-of-the-art literature known as medical image registration using deep neural networks is presented. The review is systematic and encompasses all the related works previously published in the field. Key concepts, statistical analysis from different points of view, confiding challenges, novelties and main contributions, key-enabling techniques, future directions and prospective trends all are discussed and surveyed in details in this comprehensive review. This review allows a deep understanding and insight for the readers active in the field who are investigating the state-of-the-art and seeking to contribute the future literature.

[1]  René Werner,et al.  GDL-FIRE ^\text 4D : Deep Learning-Based Fast 4D CT Image Registration , 2018, MICCAI.

[2]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Kebin Jia,et al.  Deep Adaptive Log-Demons: Diffeomorphic Image Registration with Very Large Deformations , 2015, Comput. Math. Methods Medicine.

[4]  Nilanjan Ray,et al.  Deep deformable registration: Enhancing accuracy by fully convolutional neural net , 2016, Pattern Recognit. Lett..

[5]  Yuanjie Zheng,et al.  Deep Group-Wise Registration for Multi-Spectral Images From Fundus Images , 2019, IEEE Access.

[6]  Rui Liao,et al.  Dilated FCN for Multi-Agent 2D/3D Medical Image Registration , 2017, AAAI.

[7]  Parmeet S. Bhatia,et al.  Real time coarse orientation detection in MR scans using multi-planar deep convolutional neural networks , 2017, Medical Imaging.

[8]  Nasir M. Rajpoot,et al.  Deep Autoencoder Features for Registration of Histology Images , 2018, MIUA.

[9]  Mattias P. Heinrich,et al.  Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients , 2018, International Journal of Computer Assisted Radiology and Surgery.

[10]  Sheng Xu,et al.  Adversarial Image Registration with Application for MR and TRUS Image Fusion , 2018, MLMI@MICCAI.

[11]  Josien P. W. Pluim,et al.  Supervised local error estimation for nonlinear image registration using convolutional neural networks , 2017, Medical Imaging.

[12]  Frédéric Jurie,et al.  Learning Visual Similarity Measures for Comparing Never Seen Objects , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Marc Niethammer,et al.  Quicksilver: Fast predictive image registration – A deep learning approach , 2017, NeuroImage.

[14]  Marius Staring,et al.  Robust contour propagation using deep learning and image registration for online adaptive proton therapy of prostate cancer , 2019, Medical physics.

[15]  Mert R. Sabuncu,et al.  An Unsupervised Learning Model for Deformable Medical Image Registration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Z. Jane Wang,et al.  Convolutional Neural Networks for Robust and Real-Time 2-D/3-D Registration , 2017, Deep Learning for Medical Image Analysis.

[17]  Deniz Erdogmus,et al.  Real-Time Deep Pose Estimation With Geodesic Loss for Image-to-Template Rigid Registration , 2018, IEEE Transactions on Medical Imaging.

[18]  Ben Glocker,et al.  On the Adaptability of Unsupervised CNN-Based Deformable Image Registration to Unseen Image Domains , 2018, MLMI@MICCAI.

[19]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[20]  Shu Liao,et al.  Unsupervised Deep Feature Learning for Deformable Registration of MR Brain Images , 2013, MICCAI.

[21]  Max A. Viergever,et al.  A deep learning framework for unsupervised affine and deformable image registration , 2018, Medical Image Anal..

[22]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[23]  Jean Duchon,et al.  Splines minimizing rotation-invariant semi-norms in Sobolev spaces , 1976, Constructive Theory of Functions of Several Variables.

[24]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[25]  Mattias P. Heinrich,et al.  3D-CNNs for Deep Binary Descriptor Learning in Medical Volume Data , 2018, Bildverarbeitung für die Medizin.

[26]  João Manuel R S Tavares,et al.  Medical image registration: a review , 2014, Computer methods in biomechanics and biomedical engineering.

[27]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Li Zhang,et al.  Deep similarity learning for multimodal medical images , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[29]  Dinggang Shen,et al.  Deformable Image Registration Using a Cue-Aware Deep Regression Network , 2018, IEEE Transactions on Biomedical Engineering.

[30]  Z. Jane Wang,et al.  A CNN Regression Approach for Real-Time 2D/3D Registration , 2016, IEEE Transactions on Medical Imaging.

[31]  Nikos Paragios,et al.  Linear and Deformable Image Registration with 3D Convolutional Neural Networks , 2018, RAMBO+BIA+TIA@MICCAI.

[32]  Yong Fan,et al.  Non-rigid image registration using self-supervised fully convolutional networks without training data , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[33]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[34]  Zhijian Song,et al.  Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks , 2018, Medical & Biological Engineering & Computing.

[35]  Theo van Walsum,et al.  Towards Robust CT-Ultrasound Registration Using Deep Learning Methods , 2018, MLCN/DLF/iMIMIC@MICCAI.

[36]  Eldar abanoviè,et al.  Deep Neural Network-based Feature Descriptor for Retinal Image Registration , 2018, 2018 IEEE 6th Workshop on Advances in Information, Electronic and Electrical Engineering (AIEEE).

[37]  Jing Hu,et al.  Robust Multimodal Image Registration Using Deep Recurrent Reinforcement Learning , 2018, ACCV.

[38]  Tom Vercauteren,et al.  Diffeomorphic demons: Efficient non-parametric image registration , 2009, NeuroImage.

[39]  Martin Styner,et al.  Fast predictive multimodal image registration , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[40]  Li Sun,et al.  Deformable MRI-Ultrasound Registration Using 3D Convolutional Neural Network , 2018, POCUS/BIVPCS/CuRIOUS/CPM@MICCAI.

[41]  V.R.S Mani,et al.  Survey of Medical Image Registration , 2013 .

[42]  Rui Liao,et al.  Pairwise domain adaptation module for CNN-based 2-D/3-D registration , 2018, Journal of medical imaging.

[43]  Dorin Comaniciu,et al.  An Artificial Agent for Robust Image Registration , 2016, AAAI.

[44]  Hervé Delingette,et al.  Learning a Probabilistic Model for Diffeomorphic Registration , 2018, IEEE Transactions on Medical Imaging.

[45]  Marc Modat,et al.  Label-driven weakly-supervised learning for multimodal deformarle image registration , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[46]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[47]  J. Paul Siebert,et al.  Learning Rigid Image Registration - Utilizing Convolutional Neural Networks for Medical Image Registration , 2018, BIOIMAGING.

[48]  Maxime Sermesant,et al.  SVF-Net: Learning Deformable Image Registration Using Shape Matching , 2017, MICCAI.

[49]  Yang Lei,et al.  LungRegNet: an unsupervised deformable image registration method for 4D-CT lung. , 2020, Medical physics.

[50]  Simon Lucey,et al.  Inverse Compositional Spatial Transformer Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[52]  Hervé Delingette,et al.  Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration , 2018, DLMIA/ML-CDS@MICCAI.

[53]  Mert R. Sabuncu,et al.  VoxelMorph: A Learning Framework for Deformable Medical Image Registration , 2018, IEEE Transactions on Medical Imaging.

[54]  Won-Ki Jeong,et al.  ssEMnet: Serial-Section Electron Microscopy Image Registration Using a Spatial Transformer Network with Learned Features , 2017, DLMIA/ML-CDS@MICCAI.

[55]  Stefan Heldmann,et al.  Unsupervised learning for large motion thoracic CT follow-up registration , 2019, Medical Imaging: Image Processing.

[56]  Rui Liao,et al.  Learning CNNs with Pairwise Domain Adaption for Real-Time 6DoF Ultrasound Transducer Detection and Tracking from X-Ray Images , 2017, MICCAI.

[57]  Stefan Heldmann,et al.  Enhancing Label-Driven Deep Deformable Image Registration with Local Distance Metrics for State-of-the-Art Cardiac Motion Tracking , 2018, Bildverarbeitung für die Medizin.

[58]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[59]  Dinggang Shen,et al.  Image registration by local histogram matching , 2007, Pattern Recognit..

[60]  Tao Huang,et al.  PCANet-Based Structural Representation for Nonrigid Multimodal Medical Image Registration , 2018, Sensors.

[61]  Tom Schaul,et al.  Dueling Network Architectures for Deep Reinforcement Learning , 2015, ICML.

[62]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[63]  Fumihiko Ino,et al.  An Automated Method for Generating Training Sets for Deep Learning based Image Registration , 2018, BIOIMAGING.

[64]  Xu Han,et al.  Registration of Pathological Images , 2016, SASHIMI@MICCAI.

[65]  Sarang C. Joshi,et al.  Real-Time 2D-3D Deformable Registration with Deep Learning and Application to Lung Radiotherapy Targeting , 2019, IPMI.

[66]  K. Cleary,et al.  Image-guided interventions : technology and applications , 2008 .

[67]  Nikos Komodakis,et al.  A Deep Metric for Multimodal Registration , 2016, MICCAI.

[68]  Xi Chen,et al.  An unsupervised network for fast microscopic image registration , 2018, Medical Imaging.

[69]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Diego H. Milone,et al.  Learning Deformable Registration of Medical Images with Anatomical Constraints , 2020, Neural Networks.

[72]  Nilanjan Ray,et al.  Unsupervised deformable image registration with fully connected generative neural network , 2018 .

[73]  Z. Jane Wang,et al.  Real-time 2D/3D registration via CNN regression , 2015, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[74]  Zhe-Ming Lu,et al.  Multimodal medical image registration via common representations learning and differentiable geometric constraints , 2019, Electronics Letters.

[75]  Ke Chen,et al.  An Unsupervised Deep Learning Method for Diffeomorphic Mono-and Multi-modal Image Registration , 2019, MIUA.

[76]  D. Hill,et al.  Medical image registration , 2001, Physics in medicine and biology.

[77]  Heinz Handels,et al.  Training CNNs for Image Registration from Few Samples with Model-based Data Augmentation , 2017, MICCAI.

[78]  Sheng Xu,et al.  Learning deep similarity metric for 3D MR–TRUS image registration , 2018, International Journal of Computer Assisted Radiology and Surgery.

[79]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[80]  Dinggang Shen,et al.  Adversarial Similarity Network for Evaluating Image Alignment in Deep Learning Based Registration , 2018, MICCAI.

[81]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[82]  Max A. Viergever,et al.  End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network , 2017, DLMIA/ML-CDS@MICCAI.

[83]  Andreas Maier,et al.  Metric-Driven Learning of Correspondence Weighting for 2-D/3-D Image Registration , 2018, GCPR.

[84]  Yuanjie Zheng,et al.  DGR-Net: Deep Groupwise Registration of Multispectral Images , 2019, IPMI.

[85]  Sébastien Ourselin,et al.  Weakly-supervised convolutional neural networks for multimodal image registration , 2018, Medical Image Anal..

[86]  Hongbin Zha,et al.  Non-rigid Craniofacial 2D-3D Registration Using CNN-Based Regression , 2017, DLMIA/ML-CDS@MICCAI.

[87]  Xinjian Chen,et al.  Adversarial learning for deformable registration of brain MR image using a multi-scale fully convolutional network , 2019, Biomed. Signal Process. Control..

[88]  Dinggang Shen,et al.  Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity , 2018, MLMI@MICCAI.

[89]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[90]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[91]  Nikos Paragios,et al.  Deformable Medical Image Registration: A Survey , 2013, IEEE Transactions on Medical Imaging.

[92]  Jiangping Wang,et al.  Multimodal Image Registration with Deep Context Reinforcement Learning , 2017, MICCAI.

[93]  Boudewijn P. F. Lelieveldt,et al.  Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks , 2017, MICCAI.

[94]  Nikos Paragios,et al.  Slice-to-volume medical image registration: a survey , 2017, Medical Image Anal..

[95]  Xiangrong Zhou,et al.  Learning 3D non-rigid deformation based on an unsupervised deep learning for PET/CT image registration , 2019, Medical Imaging.

[96]  Jiwen Lu,et al.  PCANet: A Simple Deep Learning Baseline for Image Classification? , 2014, IEEE Transactions on Image Processing.

[97]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[98]  Nassir Navab,et al.  Precise Ultrasound Bone Registration with Learning-Based Segmentation and Speed of Sound Calibration , 2017, MICCAI.

[99]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[100]  Paul Babyn,et al.  Generative Adversarial Network in Medical Imaging: A Review , 2018, Medical Image Anal..

[101]  Kawal S. Rhode,et al.  3D/2D model-to-image registration by imitation learning for cardiac procedures , 2018, International Journal of Computer Assisted Radiology and Surgery.

[102]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[103]  Dinggang Shen,et al.  BIRNet: Brain image registration using dual‐supervised fully convolutional networks , 2018, Medical Image Anal..

[104]  Dimitre Hristov,et al.  Feasibility of Image Registration for Ultrasound-Guided Prostate Radiotherapy Based on Similarity Measurement by a Convolutional Neural Network , 2019, Technology in cancer research & treatment.

[105]  Arthur Ardeshir Goshtasby.  Theory and Applications of Image Registration , 2017 .

[106]  Mert R. Sabuncu,et al.  Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration , 2018, MICCAI.

[107]  Yash Goyal,et al.  CloudCV: Large-Scale Distributed Computer Vision as a Cloud Service , 2015, Mobile Cloud Visual Media Computing.

[108]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[109]  Honglak Lee,et al.  Learning to Align from Scratch , 2012, NIPS.

[110]  Mitko Veta,et al.  Deformable image registration using convolutional neural networks , 2018, Medical Imaging.

[111]  Matthew T. Freedman,et al.  Computer-assisted diagnosis for lung nodule detection using a neural network technique , 1992, Medical Imaging.

[112]  Dinggang Shen,et al.  Scalable High-Performance Image Registration Framework by Unsupervised Deep Feature Representations Learning , 2016, IEEE Transactions on Biomedical Engineering.

[113]  Qian Wang,et al.  Deformable Image Registration Based on Similarity-Steered CNN Regression , 2017, MICCAI.