Improving T1w MRI-based brain tumor segmentation using cross-modal distillation

Although multi-modal imaging tends to improve the segmentation and classification performance in the field of medical image processing, lacking certain modalities at test time limits its clinical applicability. In this paper, we explored the ability of cross-modal distillation for increasing the performance of T1w MRI-based brain tumor segmentation. More specifically, we considered having high resolution T1w and T2w MRI sequences available for training while having only a high resolution T1w MRI sequence available at test time. We investigated the efficacy of the proposed method to improve the whole tumor segmentation using the BRATS 2018 dataset. Both cross-modal knowledge distillation and cross-modal feature distillation approaches were confirmed to enrich the representation of the T1w MRI sequence by learning from the representation of the more informative T2w MRI sequence during training, thereby improving the mean Dice scores by 6.14 % and 7.02 %, respectively.

[1]  Matthew B. Blaschko,et al.  Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice , 2019, MICCAI.

[2]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[3]  Tony X. Han,et al.  Learning Efficient Object Detection Models with Knowledge Distillation , 2017, NIPS.

[4]  Christos Davatzikos,et al.  Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features , 2017, Scientific Data.

[5]  Andriy Myronenko,et al.  3D MRI brain tumor segmentation using autoencoder regularization , 2018, BrainLes@MICCAI.

[6]  Raymond Y Huang,et al.  Consensus recommendations for a standardized brain tumor imaging protocol for clinical trials in brain metastases (BTIP-BM). , 2020, Neuro-oncology.

[7]  Karolien Goffin,et al.  Convolutional Neural Networks for Brain Tumor Segmentation Using Different Sets of MRI Sequences , 2019, 2019 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC).

[8]  Mohammad Havaei,et al.  HeMIS: Hetero-Modal Image Segmentation , 2016, MICCAI.

[9]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[10]  Vittorio Murino,et al.  Modality Distillation with Multiple Stream Networks for Action Recognition , 2018, ECCV.

[11]  Jaime S. Cardoso,et al.  Deep Learning and Data Labeling for Medical Applications , 2016, Lecture Notes in Computer Science.

[12]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[13]  Yoshua Bengio,et al.  FitNets: Hints for Thin Deep Nets , 2014, ICLR.

[14]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[15]  William H. Sanders,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2014 .

[16]  Trevor Darrell,et al.  Learning with Side Information through Modality Hallucination , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Klaus H. Maier-Hein,et al.  No New-Net , 2018, 1809.10483.

[18]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[19]  Rich Caruana,et al.  Do Deep Nets Really Need to be Deep? , 2013, NIPS.

[20]  Jitendra Malik,et al.  Cross Modal Distillation for Supervision Transfer , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Pheng Ann Heng,et al.  Unpaired Multi-Modal Segmentation via Knowledge Distillation , 2020, IEEE Transactions on Medical Imaging.

[22]  Parashkev Nachev,et al.  PIMMS: Permutation Invariant Multi-Modal Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[23]  Rainer Stiefelhagen,et al.  CNN-based sensor fusion techniques for multimodal human activity recognition , 2017, SEMWEB.

[24]  Marc Modat,et al.  Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation , 2019, MICCAI.

[25]  Rich Caruana,et al.  Model compression , 2006, KDD '06.