Fusion based on attention mechanism and context constraint for multi-modal brain tumor segmentation

This paper presents a 3D brain tumor segmentation network from multi-sequence MRI datasets based on deep learning. We propose a three-stage network: generating constraints, fusion under constraints and final segmentation. In the first stage, an initial 3D U-Net segmentation network is introduced to produce an additional context constraint for each tumor region. Under the obtained constraint, multi-sequence MRI are then fused using an attention mechanism to achieve three single tumor region segmentations. Considering the location relationship of the tumor regions, a new loss function is introduced to deal with the multiple class segmentation problem. Finally, a second 3D U-Net network is applied to combine and refine the three single prediction results. In each stage, only 8 initial filters are used, allowing to decrease significantly the number of parameters to be estimated. We evaluated our method on BraTS 2017 dataset. The results are promising in terms of dice score, hausdorff distance, and the amount of memory required for training.

[1]  Su Ruan,et al.  Segmentation of heterogeneous or small FDG PET positive tissue based on a 3D-locally adaptive random walk algorithm , 2014, Comput. Medical Imaging Graph..

[2]  Konstantinos Kamnitsas,et al.  Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation , 2017, BrainLes@MICCAI.

[3]  Chunfeng Lian,et al.  Joint Tumor Segmentation in PET-CT Images Using Co-Clustering and Fusion Based on Belief Functions , 2019, IEEE Transactions on Image Processing.

[4]  Changsheng Li,et al.  Multimodal brain tumor image segmentation using WRN-PPNet , 2019, Comput. Medical Imaging Graph..

[5]  Ben Glocker,et al.  Decision Forests for Tissue-Specific Segmentation of High-Grade Gliomas in Multi-channel MR , 2012, MICCAI.

[6]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[7]  Cem Direkoglu,et al.  Review of MRI-based Brain Tumor Image Segmentation Using Deep Learning Methods , 2016 .

[8]  Victor Alves,et al.  Adaptive feature recombination and recalibration for semantic segmentation: application to brain tumor segmentation in MRI , 2018, MICCAI.

[9]  Kuan-Lun Tseng,et al.  Joint Sequence Learning and Cross-Modality Convolution for 3D Biomedical Segmentation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  S. Bauer,et al.  A survey of MRI-based medical image analysis for brain tumor studies , 2013, Physics in medicine and biology.

[11]  Paul C. Lauterbur,et al.  Principles of magnetic resonance imaging : a signal processing perspective , 1999 .

[12]  Tal Arbel,et al.  Brain Tumor Segmentation Using a 3D FCN with Multi-scale Loss , 2017, BrainLes@MICCAI.

[13]  Chang Liu,et al.  Automatic Semantic Segmentation of Brain Gliomas from MRI Images Using a Deep Cascaded Neural Network , 2018, Journal of healthcare engineering.

[14]  Su Ruan,et al.  A review: Deep learning for medical image segmentation using multi-modality fusion , 2019, Array.

[15]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[16]  Yong Fan,et al.  A deep learning model integrating FCNNs and CRFs for brain tumor segmentation , 2017, Medical Image Anal..

[17]  Stefan Bauer,et al.  Fully Automatic Segmentation of Brain Tumor Images Using Support Vector Machine Classification in Combination with Hierarchical Conditional Random Field Regularization , 2011, MICCAI.

[18]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Su Ruan,et al.  Semi-automatic lymphoma detection and segmentation using fully conditional random fields , 2018, Comput. Medical Imaging Graph..

[20]  Su Ruan,et al.  Deep Learning Model Integrating Dilated Convolution and Deep Supervision for Brain Tumor Segmentation in Multi-parametric MRI , 2019, MLMI@MICCAI.

[21]  Nassir Navab,et al.  Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks , 2018, MICCAI.

[22]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[23]  Antonios Drevelegas.  Imaging Of Brain Tumors With Histological Correlations , 2002 .

[24]  Klaus H. Maier-Hein,et al.  Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge , 2017, BrainLes@MICCAI.

[25]  Hervé Delingette,et al.  3D Convolutional Neural Networks for Tumor Segmentation using Long-range 2D Context , 2018, Comput. Medical Imaging Graph..

[26]  Su Ruan,et al.  Segmenting Multi-Source Images Using Hidden Markov Fields With Copula-Based Multivariate Statistical Distributions , 2017, IEEE Transactions on Image Processing.

[27]  Jun Fu,et al.  Dual Attention Network for Scene Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Nan Zhang,et al.  Kernel feature selection to fuse multi-spectral MRI images for brain tumor segmentation , 2011, Comput. Vis. Image Underst..

[29]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[30]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[31]  Christos Davatzikos,et al.  Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features , 2017, Scientific Data.

[32]  Sébastien Ourselin,et al.  Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks , 2017, BrainLes@MICCAI.

[33]  Wolfram Burgard,et al.  Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion , 2016, ISER.