Transfer Brain MRI Tumor Segmentation Models Across Modalities with Adversarial Networks

In this work, we present an approach to brain cancer segmentation in Magnetic Resonance Images (MRI) using Adversarial Networks, that have been successfully applied to several complex image processing problems in recent years. Most of the segmentation approaches presented in the literature exploit the data from all the contrast modalities typically acquired in the clinical practice: T1-weighted, T1-weighted contrast-enhanced, T2-weighted, and T2-FLAIR. Unfortunately, often not all these modalities are available for each patient. Accordingly, in this paper, we extended a previous segmentation approach based on Adversarial Networks to deal with this issue. In particular, we trained a segmentation model for each modality at once and evaluated the performances of these models. Thus, we investigated the possibility of transferring the best among these single-modality models to the other modalities. Our results suggest that such a transfer learning approach allows achieving better performances for almost all the target modalities.

[1]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .

[2]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[3]  Mohammad Havaei,et al.  HeMIS: Hetero-Modal Image Segmentation , 2016, MICCAI.

[4]  Isabelle Bloch,et al.  From neonatal to adult brain MR image segmentation in a few seconds using 3D-like fully convolutional network and transfer learning , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[5]  Camille Couprie,et al.  Semantic Segmentation using Adversarial Networks , 2016, NIPS 2016.

[6]  Ghassan Hamarneh,et al.  Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network , 2019, IEEE Transactions on Medical Imaging.

[7]  Bogdan Raducanu,et al.  Transferring GANs: generating images from limited data , 2018, ECCV.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  Max A. Viergever,et al.  Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities , 2016, MICCAI.

[10]  Mitko Veta,et al.  Adversarial Training and Dilated Convolutions for Brain MRI Segmentation , 2017, DLMIA/ML-CDS@MICCAI.

[11]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  Yaël Frégier,et al.  Geometric Science of Information: 5th International Conference, GSI 2021, Paris, France, July 21–23, 2021, Proceedings , 2019, GSI.

[13]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[14]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[15]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[16]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[17]  Lei Wang,et al.  3D cGAN based cross-modality MR image synthesis for brain tumor segmentation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[18]  Tolga Çukur,et al.  A Transfer‐Learning Approach for Accelerated MRI Using Deep Neural Networks , 2017, Magnetic resonance in medicine.

[19]  Irene Y. H. Gu,et al.  Cross-Modality Augmentation of Brain Mr Images Using a Novel Pairwise Generative Adversarial Network for Enhanced Glioma Classification , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[20]  Masoom A. Haider,et al.  A Transfer Learning Approach for Automated Segmentation of Prostate Whole Gland and Transition Zone in Diffusion Weighted MRI , 2019, ArXiv.

[21]  Joseph O. Deasy,et al.  Tumor-Aware, Adversarial Domain Adaptation from CT to MRI for Lung Cancer Segmentation , 2018, MICCAI.

[22]  Tao Xu,et al.  SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation , 2017, Neuroinformatics.

[23]  Nico Karssemeijer,et al.  Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation , 2017, MICCAI.

[24]  M. Jorge Cardoso,et al.  Simultaneous synthesis of FLAIR and segmentation of white matter hypointensities from T1 MRIs , 2018, ArXiv.

[25]  Parashkev Nachev,et al.  PIMMS: Permutation Invariant Multi-Modal Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[26]  Marc Modat,et al.  Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation , 2019, MICCAI.

[27]  Aykut Erdem,et al.  Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks , 2018, IEEE Transactions on Medical Imaging.

[28]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).