LSTM Spatial Co-transformer Networks for Registration of 3D Fetal US and MR Brain Images

In this work, we propose a deep learning-based method for iterative registration of fetal brain images acquired by ultrasound and magnetic resonance, inspired by “Spatial Transformer Networks”. Images are co-aligned to a dual modality spatio-temporal atlas, where computational image analysis may be performed in the future. Our results show better alignment accuracy compared to “Self-Similarity Context descriptors”, a state-of-the-art method developed for multi-modal image registration. Furthermore, our method is robust and able to register highly misaligned images, with any initial orientation, where similarity-based methods typically fail.

[1]  Michael Brady,et al.  Towards Realtime Multimodal Fusion for Image-Guided Interventions Using Self-similarities , 2013, MICCAI.

[2]  Mert R. Sabuncu,et al.  An Unsupervised Learning Model for Deformable Medical Image Registration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Michael Brady,et al.  Phase mutual information as a similarity measure for registration , 2005, Medical Image Anal..

[4]  Nikos Komodakis,et al.  A Deep Metric for Multimodal Registration , 2016, MICCAI.

[5]  Sébastien Ourselin,et al.  Reconstructing a 3D structure from serial histological sections , 2001, Image Vis. Comput..

[6]  J. Alison Noble,et al.  Registration of 3D Fetal Brain US and MRI , 2012, MICCAI.

[7]  Jesús Chamorro-Martínez,et al.  Diatom autofocusing in brightfield microscopy: a comparative study , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[8]  Maxime Sermesant,et al.  SVF-Net: Learning Deformable Image Registration Using Shape Matching , 2017, MICCAI.

[9]  Nassir Navab,et al.  Entropy and Laplacian images: Structural representations for multi-modal registration , 2012, Medical Image Anal..

[10]  Simon Lucey,et al.  Inverse Compositional Spatial Transformer Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Eldad Haber,et al.  Intensity Gradient Based Registration and Fusion of Multi-modal Images , 2006, MICCAI.

[12]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[13]  D. Louis Collins,et al.  Self-similarity weighted mutual information: A new nonrigid image registration metric , 2014, Medical Image Anal..

[14]  Daniel Rueckert,et al.  Construction of a consistent high-definition spatio-temporal atlas of the developing brain using adaptive kernel regression , 2012, NeuroImage.

[15]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.