Weakly-supervised convolutional neural networks for multimodal image registration

HighlightsA method to infer voxel‐level correspondence from higher‐level anatomical labels.Efficient and fully‐automated registration for MR and ultrasound prostate images.Validation experiments with 108 pairs of labelled interventional patient images.Open‐source implementation. ABSTRACT One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground‐truth for voxel‐level spatial correspondence. This work describes a method to infer voxel‐level transformation from higher‐level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel‐level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject‐specific ad hoc landmarks. The proposed end‐to‐end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real‐time and is fully‐automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2‐weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross‐validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high‐quality anatomical labels. Graphical abstract Figure. No caption available.

[1]  Shu Liao,et al.  Unsupervised Deep Feature Learning for Deformable Registration of MR Brain Images , 2013, MICCAI.

[2]  Aaron Fenster,et al.  Robust 2-D–3-D Registration Optimization for Motion Compensation During 3-D TRUS-Guided Biopsy Using Learned Prostate Motion Data , 2017, IEEE Transactions on Medical Imaging.

[3]  J. Alison Noble.  Reflections on ultrasound image analysis , 2016, Medical Image Anal..

[4]  Pheng-Ann Heng,et al.  Patient-specific Deformation Modelling via Elastography: Application to Image-guided Prostate Interventions , 2016, Scientific Reports.

[5]  D. Hawkes,et al.  Microstructural Characterization of Normal and Malignant Human Prostate Tissue With Vascular, Extracellular, and Restricted Diffusion for Cytometry in Tumours Magnetic Resonance Imaging , 2015, Investigative radiology.

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Maxime Sermesant,et al.  SVF-Net: Learning Deformable Image Registration Using Shape Matching , 2017, MICCAI.

[8]  J. Modersitzki,et al.  A unified approach to fast image registration and a new curvature based registration technique , 2004 .

[9]  Nikos Komodakis,et al.  A Deep Metric for Multimodal Registration , 2016, MICCAI.

[10]  Max A. Viergever,et al.  A survey of medical image registration - under review , 2016, Medical Image Anal..

[11]  Dean C. Barratt,et al.  Towards Image-Guided Pancreas and Biliary Endoscopy: Automatic Multi-organ Segmentation on Abdominal CT with Dense Dilated Networks , 2017, MICCAI.

[12]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[13]  Dean C. Barratt,et al.  Automatic slice segmentation of intraoperative transrectal ultrasound images using convolutional neural networks , 2018, Medical Imaging.

[14]  Konstantinos G. Derpanis,et al.  Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness , 2016, ECCV Workshops.

[15]  Dean C. Barratt,et al.  Population-based prediction of subject-specific prostate deformation for MR-to-ultrasound image registration , 2015, Medical Image Anal..

[16]  Baris Turkbey,et al.  Comparison of MR/ultrasound fusion-guided biopsy with ultrasound-guided biopsy for the diagnosis of prostate cancer. , 2015, JAMA.

[17]  John W. Fisher,et al.  A Unified Statistical and Information Theoretic Framework for Multi-modal Image Registration , 2003, IPMI.

[18]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Aaron Fenster,et al.  Three-Dimensional Nonrigid MR-TRUS Registration Using Dual Optimization , 2015, IEEE Transactions on Medical Imaging.

[20]  John Kurhanewicz,et al.  Hyperpolarized 13C MR for Molecular Imaging of Prostate Cancer , 2014, The Journal of Nuclear Medicine.

[21]  Qian Wang,et al.  Deformable Image Registration Based on Similarity-Steered CNN Regression , 2017, MICCAI.

[22]  H. Hricak,et al.  Updated prostate imaging reporting and data system (PIRADS v2) recommendations for the detection of clinically significant prostate cancer using multiparametric MRI: critical evaluation using whole-mount pathology as standard of reference , 2016, European Radiology.

[23]  Nico Karssemeijer,et al.  Surface-based prostate registration with biomechanical regularization , 2013, Medical Imaging.

[24]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[25]  B. Carey,et al.  Magnetic resonance imaging for the detection, localisation, and characterisation of prostate cancer: recommendations from a European consensus meeting. , 2011, European urology.

[26]  Mrityunjay Kumar,et al.  A Total Variation-Based Algorithm for Pixel-Level Image Fusion , 2009, IEEE Transactions on Image Processing.

[27]  Purang Abolmaesumi,et al.  Statistical Biomechanical Surface Registration: Application to MR-TRUS Fusion for Prostate Interventions , 2015, IEEE Transactions on Medical Imaging.

[28]  Marc Niethammer,et al.  Quicksilver: Fast predictive image registration – A deep learning approach , 2017, NeuroImage.

[29]  Sébastien Ourselin,et al.  Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations , 2017, DLMIA/ML-CDS@MICCAI.

[30]  Daniel Rueckert,et al.  Nonrigid registration using free-form deformations: application to breast MR images , 1999, IEEE Transactions on Medical Imaging.

[31]  Dorin Comaniciu,et al.  An Artificial Agent for Robust Image Registration , 2016, AAAI.

[32]  Sébastien Ourselin,et al.  Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning , 2017, IEEE Transactions on Medical Imaging.

[33]  J. L. Chin,et al.  Urogenital Ultrasound: A Text Atlas, 2nd ed, D.L. Cochlin, P.A. Dubbins, B.B. Goldberg, E.J. Halpern. Taylor & Francis, Boca Raton, Florida (2006), 464 pages , 2007 .

[34]  Parashkev Nachev,et al.  Computer Methods and Programs in Biomedicine NiftyNet: a deep-learning platform for medical imaging , 2022 .

[35]  Sébastien Ourselin,et al.  ToolNet: Holistically-nested real-time segmentation of robotic surgical tools , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[36]  Max A. Viergever,et al.  A survey of medical image registration , 1998, Medical Image Anal..

[37]  Hao Chen,et al.  Volumetric ConvNets with Mixed Residual Connections for Automated Prostate Segmentation from 3D MR Images , 2017, AAAI.

[38]  Baris Turkbey,et al.  Improving detection of clinically significant prostate cancer: magnetic resonance imaging/transrectal ultrasound fusion guided prostate biopsy. , 2014, The Journal of urology.

[39]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[40]  Nico Karssemeijer,et al.  Biomechanical modeling constrained surface-based image registration for prostate MR guided TRUS biopsy. , 2015, Medical physics.

[41]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[42]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Charles Elkan,et al.  The Foundations of Cost-Sensitive Learning , 2001, IJCAI.

[44]  Daniel Rueckert,et al.  Registration Using Sparse Free-Form Deformations , 2012, MICCAI.

[45]  Ming Xu,et al.  Towards Personalized Statistical Deformable Model and Hybrid Point Matching for Robust MR-TRUS Registration , 2016, IEEE Transactions on Medical Imaging.

[46]  Balraj Naren,et al.  Medical Image Registration , 2022 .

[47]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[48]  Zhuowen Tu,et al.  Deeply-Supervised Nets , 2014, AISTATS.

[49]  Haiying Liu,et al.  A Generic Framework for Non-rigid Registration Based on Non-uniform Multi-level Free-Form Deformations , 2001, MICCAI.

[50]  Max A. Viergever,et al.  End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network , 2017, DLMIA/ML-CDS@MICCAI.

[51]  Zeike A. Taylor,et al.  MR to ultrasound registration for image-guided prostate interventions , 2012, Medical Image Anal..

[52]  Geoffrey E. Hinton,et al.  Regularizing Neural Networks by Penalizing Confident Output Distributions , 2017, ICLR.

[53]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Tobias Gass,et al.  Isotropic Total Variation Regularization of Displacements in Parametric Image Registration , 2017, IEEE Transactions on Medical Imaging.

[55]  Y. Hu,et al.  Registration of magnetic resonance and ultrasound images for guiding prostate cancer interventions , 2013 .

[56]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Boudewijn P. F. Lelieveldt,et al.  Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks , 2017, MICCAI.

[58]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[59]  David J. Hawkes,et al.  Modelling Prostate Motion for Data Fusion During Image-Guided Interventions , 2011, IEEE Transactions on Medical Imaging.

[60]  Ah Chung Tsoi,et al.  Neural Network Classification and Prior Class Probabilities , 1996, Neural Networks: Tricks of the Trade.

[61]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[62]  Sergio Guadarrama,et al.  The Devil is in the Decoder , 2017, BMVC.

[63]  Z. Jane Wang,et al.  A CNN Regression Approach for Real-Time 2D/3D Registration , 2016, IEEE Transactions on Medical Imaging.

[64]  Pierre Mozer,et al.  Detection of Clinically Significant Prostate Cancer Using Magnetic Resonance Imaging-Ultrasound Fusion Targeted Biopsy: A Systematic Review. , 2015, European urology.

[65]  David J. Hawkes,et al.  Development and Phantom Validation of a 3-D-Ultrasound-Guided System for Targeting MRI-Visible Lesions During Transrectal Prostate Biopsy , 2017, IEEE Transactions on Biomedical Engineering.

[66]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  V.R.S Mani,et al.  Survey of Medical Image Registration , 2013 .

[68]  Marc Modat,et al.  Label-driven weakly-supervised learning for multimodal deformarle image registration , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[69]  Alistair A. Young,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2017, MICCAI 2017.

[70]  Hervé Delingette,et al.  Robust Non-rigid Registration Through Agent-Based Action Learning , 2017, MICCAI.

[71]  D. Hill,et al.  Medical image registration , 2001, Physics in medicine and biology.

[72]  Sébastien Ourselin,et al.  Fast free-form deformation using graphics processing units , 2010, Comput. Methods Programs Biomed..

[73]  Hao Chen,et al.  3D deeply supervised network for automated segmentation of volumetric medical images , 2017, Medical Image Anal..

[74]  Pingkun Yan,et al.  Magnetic resonance imaging/ultrasound fusion guided prostate biopsy improves cancer detection following transrectal ultrasound biopsy and correlates with multiparametric magnetic resonance imaging. , 2011, The Journal of urology.