Loss odyssey in medical image segmentation

The loss function is an important component in deep learning-based segmentation methods. Over the past five years, many loss functions have been proposed for various segmentation tasks. However, a systematic study of the utility of these loss functions is missing. In this paper, we present a comprehensive review of segmentation loss functions in an organized manner. We also conduct the first large-scale analysis of 20 general loss functions on four typical 3D segmentation tasks involving six public datasets from 10+ medical centers. The results show that none of the losses can consistently achieve the best performance on the four segmentation tasks, but compound loss functions (e.g. Dice with TopK loss, focal loss, Hausdorff distance loss, and boundary loss) are the most robust losses. Our code and segmentation results are publicly available and can serve as a loss function benchmark. We hope this work will also provide insights on new loss function development for the community.

[1]  Septimiu E. Salcudean,et al.  Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks , 2019, IEEE Transactions on Medical Imaging.

[2]  Deniz Erdogmus,et al.  Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks , 2017, MLMI@MICCAI.

[3]  Naimul Mefraz Khan,et al.  A Novel Focal Tversky Loss Function With Improved Attention U-Net for Lesion Segmentation , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[4]  Ghassan Hamarneh,et al.  Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation , 2018, Comput. Medical Imaging Graph..

[5]  Dimitris Samaras,et al.  Topology-Preserving Deep Image Segmentation , 2019, NeurIPS.

[6]  Matthew B. Blaschko,et al.  Optimization for Medical Image Segmentation: Theory and Practice When Evaluating With Dice Score or Jaccard Index , 2020, IEEE Transactions on Medical Imaging.

[7]  S. Heiland,et al.  Automated volumetric assessment with artificial neural networks might enable a more accurate assessment of disease burden in patients with multiple sclerosis , 2020, European Radiology.

[8]  Anonymous Name,et al.  How Distance Transform Maps Boost Segmentation CNNs: An Empirical Study , 2020 .

[9]  Konstantinos Kamnitsas,et al.  Analyzing Overfitting Under Class Imbalance in Neural Networks for Image Segmentation , 2020, IEEE Transactions on Medical Imaging.

[10]  Tanveer F. Syeda-Mahmood,et al.  3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes , 2018, MICCAI.

[11]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[12]  Ben Glocker,et al.  Attention Gated Networks: Learning to Leverage Salient Regions in Medical Images , 2018, Medical Image Anal..

[13]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[14]  Anvar Kurmukov,et al.  Universal Loss Reweighting to Balance Lesion Size Inequality in 3D Medical Image Segmentation , 2020, MICCAI.

[15]  Matthew B. Blaschko,et al.  Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory and Practice , 2019, MICCAI.

[16]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[17]  Xiaohui Xie,et al.  AnatomyNet: Deep learning for fast and fully automated whole‐volume segmentation of head and neck anatomy , 2018, Medical physics.

[18]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[19]  Maxime Bassenne,et al.  Closing the Gap Between Deep Neural Network Modeling and Biomedical Decision-Making Metrics in Segmentation via Adaptive Loss Functions , 2020, IEEE Transactions on Medical Imaging.

[20]  Mohammad Sohel Rahman,et al.  MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation , 2019, Neural Networks.

[21]  Xin Yang,et al.  Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved? , 2018, IEEE Transactions on Medical Imaging.

[22]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[23]  Young-Hak Kim,et al.  Major Vessel Segmentation on X-ray Coronary Angiography using Deep Networks with a Novel Penalty Loss Function , 2019 .

[24]  Sébastien Ourselin,et al.  Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations , 2017, DLMIA/ML-CDS@MICCAI.

[25]  Hao Chen,et al.  The Liver Tumor Segmentation Benchmark (LiTS) , 2019, Medical Image Anal..

[26]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[28]  Claudia Iriondo,et al.  Distance Map Loss Penalty Term for Semantic Segmentation , 2019, ArXiv.

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[31]  Simon K. Warfield,et al.  Asymmetric Loss Functions and Deep Densely-Connected Networks for Highly-Imbalanced Medical Image Segmentation: Application to Multiple Sclerosis Lesion Detection , 2018, IEEE Access.

[32]  Hugues Benoit-Cattin,et al.  Removing segmentation inconsistencies with semi-supervised non-adjacency constraint , 2019, Medical Image Anal..

[33]  Matthew B. Blaschko,et al.  The Lovasz-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Dean C. Barratt,et al.  Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks , 2018, IEEE Transactions on Medical Imaging.

[35]  Lisa Tang,et al.  Deep Convolutional Encoder Networks for Multiple Sclerosis Lesion Segmentation , 2015, MICCAI.

[36]  Hao Li,et al.  Visualizing the Loss Landscape of Neural Nets , 2017, NeurIPS.

[37]  Yang Wang,et al.  Optimizing Intersection-Over-Union in Deep Neural Networks for Image Segmentation , 2016, ISVC.

[38]  Yaozong Gao,et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge , 2019, Medical Image Anal..

[39]  Steven Guan,et al.  Fully Dense UNet for 2-D Sparse Photoacoustic Tomography Artifact Removal , 2018, IEEE Journal of Biomedical and Health Informatics.

[40]  Guang Yang,et al.  Evaluation of algorithms for Multi-Modality Whole Heart Segmentation: An open-access grand challenge , 2019, Medical Image Anal..

[41]  Sébastien Ourselin,et al.  Generalised Wasserstein Dice Score for Imbalanced Multi-class Segmentation using Holistic Convolutional Networks , 2017, BrainLes@MICCAI.

[42]  Zhiming Luo,et al.  Weighted Res-UNet for High-Quality Retina Vessel Segmentation , 2018, 2018 9th International Conference on Information Technology in Medicine and Education (ITME).

[43]  Jose Dolz,et al.  Boundary loss for highly unbalanced segmentation , 2018, MIDL.