Normalization in Training U-Net for 2-D Biomedical Semantic Segmentation

Two-dimensional (2-D) biomedical semantic segmentation is important for robotic vision in surgery. Segmentation methods based on deep convolutional neural network (DCNN) can out-perform conventional methods in terms of both accuracy and levels of automation. One common issue in training a DCNN for biomedical semantic segmentation is the internal covariate shift where the training of convolutional kernels is encumbered by the distribution change of input features, hence both the training speed and performance are decreased. Batch normalization (BN) is the first proposed method for addressing internal covariate shift and is widely used. Instance normalization (IN) and layer normalization (LN) have also been proposed. Group normalization (GN) is proposed more recently and has not yet been applied to 2-D biomedical semantic segmentation (GN was used in 3-D biomedical semantic segmentation in [P.-Y. Kao, T. Ngo, A. Zhang, J. Chen, and B. Manjunath, Brain tumor segmentation and tractographic feature extraction from structural MR images for overall survival prediction 2018, arXiv:1807.07716], however, no specific validations on GN were given). Most DCNNs for biomedical semantic segmentation adopt BN as the normalization method by default, without reviewing its performance. In this letter, four normalization methods—BN, IN, LN, and GN are compared in details, specifically for 2-D biomedical semantic segmentation. U-Net is adopted as the basic DCNN structure. Three datasets regarding the right ventricle, aorta, and left ventricle are used for the validation. The results show that detailed subdivision of the feature map, i.e., GN with a large group number or IN, achieves higher accuracy. This accuracy improvement mainly comes from better model generalization. Codes are uploaded and maintained at Xiao-Yun Zhou's Github.

[1]  Guang-Zhong Yang,et al.  Outlier Detection and Handling for Robust 3-D Active Shape Models Search , 2007, IEEE Transactions on Medical Imaging.

[2]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[3]  Su-Lin Lee,et al.  Path planning for robot-enhanced cardiac Radiofrequency Catheter Ablation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[4]  Mario Ceresa,et al.  Fully automatic detection and segmentation of abdominal aortic thrombus in post‐operative CTA images using Deep Convolutional Neural Networks , 2018, Medical Image Anal..

[5]  Tim Salimans,et al.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks , 2016, NIPS.

[6]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[7]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[8]  Su-Lin Lee,et al.  A real‐time and registration‐free framework for dynamic shape instantiation , 2017, Medical Image Anal..

[9]  Jianfeng Zhan,et al.  Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks , 2017, ICANN.

[10]  Hao Chen,et al.  3D multi‐scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multi‐modality MR Images , 2018, Medical Image Anal..

[11]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[13]  Hyo-Eun Kim,et al.  Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks , 2018, NeurIPS.

[14]  Guoyan Zheng,et al.  3 D multi-scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multimodality MR Images , 2018 .

[15]  Jun Zhang,et al.  Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI With Application to Radiogenomics , 2019, IEEE Transactions on Medical Imaging.

[16]  Liang Lin,et al.  Kalman Normalization: Normalizing Internal Representations Across Network Layers , 2018, NeurIPS.

[17]  B. S. Manjunath,et al.  Brain Tumor Segmentation and Tractographic Feature Extraction from Structural MR Images for Overall Survival Prediction , 2018, BrainLes@MICCAI.

[18]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[20]  Kaiming He,et al.  Group Normalization , 2018, ECCV.

[21]  Tobias Gass,et al.  Cloud-Based Evaluation of Anatomical Structure Segmentation and Landmark Detection Algorithms: VISCERAL Anatomy Benchmarks , 2016, IEEE Transactions on Medical Imaging.

[22]  Jichao Zhao,et al.  Fully Automatic Left Atrium Segmentation From Late Gadolinium Enhanced Magnetic Resonance Imaging Using a Dual Fully Convolutional Neural Network , 2019, IEEE Transactions on Medical Imaging.

[23]  Vivienne Sze,et al.  Efficient Processing of Deep Neural Networks: A Tutorial and Survey , 2017, Proceedings of the IEEE.

[24]  Andreas K. Maier,et al.  Adaption of 3D Models to 2D X-Ray Images during Endovascular Abdominal Aneurysm Repair , 2015, MICCAI.

[25]  Su-Lin Lee,et al.  Real-Time 3-D Shape Instantiation From Single Fluoroscopy Projection for Fenestrated Stent Graft Deployment , 2017, IEEE Robotics and Automation Letters.

[26]  Quoc V. Le,et al.  Don't Decay the Learning Rate, Increase the Batch Size , 2017, ICLR.

[27]  Su-Lin Lee,et al.  Towards Automatic 3D Shape Instantiation for Deployed Stent Grafts: 2D Multiple-class and Class-imbalance Marker Segmentation with Equally-weighted Focal U-Net , 2017, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Xin Yang,et al.  Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved? , 2018, IEEE Transactions on Medical Imaging.

[29]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..