Latent Correlation Representation Learning for Brain Tumor Segmentation With Missing MRI Modalities

Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess brain tumor. Accurately segmenting brain tumor from MR images is the key to clinical diagnostics and treatment planning. In addition, multi-modal MR images can provide complementary information for accurate brain tumor segmentation. However, it’s common to miss some imaging modalities in clinical practice. In this paper, we present a novel brain tumor segmentation algorithm with missing modalities. Since it exists a strong correlation between multi-modalities, a correlation model is proposed to specially represent the latent multi-source correlation. Thanks to the obtained correlation representation, the segmentation becomes more robust in the case of missing modality. First, the individual representation produced by each encoder is used to estimate the modality independent parameter. Then, the correlation model transforms all the individual representations to the latent multi-source correlation representations. Finally, the correlation representations across modalities are fused via attention mechanism into a shared representation to emphasize the most important features for segmentation. We evaluate our model on BraTS 2018 and BraTS 2019 dataset, it outperforms the current state-of-the-art methods and produces robust results when one or more modalities are missing.

[1]  Hervé Delingette,et al.  3D Convolutional Neural Networks for Tumor Segmentation using Long-range 2D Context , 2018, Comput. Medical Imaging Graph..

[2]  Loïc Le Folgoc,et al.  Attention U-Net: Learning Where to Look for the Pancreas , 2018, ArXiv.

[3]  Ben Glocker,et al.  Decision Forests for Tissue-Specific Segmentation of High-Grade Gliomas in Multi-channel MR , 2012, MICCAI.

[4]  Philip Chikontwe,et al.  Two-Step U-Nets for Brain Tumor Segmentation and Random Forest with Radiomics for Survival Time Prediction , 2019, BrainLes@MICCAI.

[5]  Chunfeng Lian,et al.  Joint Tumor Segmentation in PET-CT Images Using Co-Clustering and Fusion Based on Belief Functions , 2019, IEEE Transactions on Image Processing.

[6]  Nassir Navab,et al.  Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks , 2018, MICCAI.

[7]  Hans J. Johnson,et al.  Advanced Normalization Tools (ANTs) , 2020 .

[8]  Xiang Liu,et al.  Brain Tumor Segmentation on Multimodal MR Imaging Using Multi-level Upsampling in Decoder , 2018, BrainLes@MICCAI.

[9]  Pham The Bao,et al.  Brain Tumor Segmentation Using Bit-plane and UNET , 2018, BrainLes@MICCAI.

[10]  Sébastien Ourselin,et al.  Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks , 2017, BrainLes@MICCAI.

[11]  Jun Fu,et al.  Dual Attention Network for Scene Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Changsheng Li,et al.  Multimodal brain tumor image segmentation using WRN-PPNet , 2019, Comput. Medical Imaging Graph..

[13]  Nan Zhang,et al.  Kernel feature selection to fuse multi-spectral MRI images for brain tumor segmentation , 2011, Comput. Vis. Image Underst..

[14]  Jianhuang Wu,et al.  Aggregating Multi-scale Prediction Based on 3D U-Net in Brain Tumor Segmentation , 2019, BrainLes@MICCAI.

[15]  Su Ruan,et al.  Segmenting Multi-Source Images Using Hidden Markov Fields With Copula-Based Multivariate Statistical Distributions , 2017, IEEE Transactions on Image Processing.

[16]  Jonas Adler,et al.  A unified representation network for segmentation with missing modalities , 2019, ArXiv.

[17]  Joseph Chazalon,et al.  Using Separated Inputs for Multimodal Brain Tumor Segmentation with 3D U-Net-like Architectures , 2019, BrainLes@MICCAI.

[18]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[19]  Rafael C. González,et al.  Digital image processing using MATLAB , 2006 .

[20]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[21]  Hao Chen,et al.  Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion , 2019, MICCAI.

[22]  Konstantinos Kamnitsas,et al.  Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation , 2017, BrainLes@MICCAI.

[23]  Stefan Bauer,et al.  Fully Automatic Segmentation of Brain Tumor Images Using Support Vector Machine Classification in Combination with Hierarchical Conditional Random Field Regularization , 2011, MICCAI.

[24]  Mohammadreza Soltaninejad,et al.  Multi-Resolution 3D CNN for MRI Brain Tumor Segmentation and Survival Prediction , 2019, BrainLes@MICCAI.

[25]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Marc Modat,et al.  Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation , 2019, MICCAI.

[27]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[28]  Su Ruan,et al.  Semi-automatic lymphoma detection and segmentation using fully conditional random fields , 2018, Comput. Medical Imaging Graph..

[29]  Andriy Myronenko,et al.  3D MRI brain tumor segmentation using autoencoder regularization , 2018, BrainLes@MICCAI.

[30]  Yan Shen,et al.  Brain Tumor Segmentation on MRI with Missing Modalities , 2019, IPMI.

[31]  Tianfu Wang,et al.  Brain Tumor Synthetic Segmentation in 3D Multimodal MRI Scans , 2019, BrainLes@MICCAI.

[32]  Chang Liu,et al.  Automatic Semantic Segmentation of Brain Gliomas from MRI Images Using a Deep Cascaded Neural Network , 2018, Journal of healthcare engineering.

[33]  Rupal Agravat,et al.  Brain Tumor Segmentation and Survival Prediction , 2019, BrainLes@MICCAI.

[34]  Nassir Navab,et al.  'Squeeze & Excite' Guided Few-Shot Segmentation of Volumetric Images , 2019, Medical Image Anal..

[35]  Pengfei Xiong,et al.  Pyramid Attention Network for Semantic Segmentation , 2018, BMVC.

[36]  Qiule Sun,et al.  Memory-Efficient Cascade 3D U-Net for Brain Tumor Segmentation , 2019, BrainLes@MICCAI.

[37]  Mohammad Havaei,et al.  HeMIS: Hetero-Modal Image Segmentation , 2016, MICCAI.

[38]  Dacheng Tao,et al.  Two-Stage Cascaded U-Net: 1st Place Solution to BraTS Challenge 2019 Segmentation Task , 2019, BrainLes@MICCAI.

[39]  Klaus H. Maier-Hein,et al.  Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge , 2017, BrainLes@MICCAI.

[40]  Su Ruan,et al.  A review: Deep learning for medical image segmentation using multi-modality fusion , 2019, Array.

[41]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[42]  Dacheng Tao,et al.  One-Pass Multi-Task Networks With Cross-Task Guided Attention for Brain Tumor Segmentation , 2019, IEEE Transactions on Image Processing.