Multi-class multimodal semantic segmentation with an improved 3D fully convolutional networks

Abstract Semantic segmentation is an important but challenging task in the field of medical image analysis. Automatic labeling for different anatomical structures can be useful for disease diagnosis, treatment planning and development/degeneration evaluation. However, due to the large shape and appearance variance among different subjects, accurate and reliable semantic segmentation is difficult for automatic labeling and time-consuming for manual labeling. In this paper, we propose a multi-class semantic segmentation algorithm based on the 3D fully convolutional networks, exploring the multimodal image blocks. The use of multimodal image blocks naturally allows an effective data augmentation. Besides, we also investigated the use of ensemble learning and conditional random fields (CRFs) as the post-processing step, which obtains the spatially consistent segmentation results. Our proposed method was evaluated on two public benchmarks: BRATS2017 and MNI-HiSUB25. For both datasets, the experimental results demonstrated that our method was able to establish an effective semantic segmentation framework with multi-modality data and achieve better performance compared to the conventional fully convolutional network.