Trimmed Robust Loss Function for Training Deep Neural Networks with Label Noise

Deep neural networks obtain nowadays outstanding results on many vision, speech recognition and natural language processing-related tasks. Such deep structures need to be trained on very large datasets, what makes annotating the data for supervised learning, particularly difficult and time-consuming task. In the supervised datasets label noise may occur, which makes the whole training process less reliable. In this paper we present a novel robust loss function based on categorical cross-entropy. We demonstrate its robustness for several amounts of noisy labels, on popular MNIST and CIFAR-10 datasets.

[1]  Xiaogang Wang,et al.  Learning from massive noisy labeled data for image classification , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Andrzej Rusiecki,et al.  Reducing noise impact on MLP training , 2016, Soft Comput..

[3]  A. Rusiecki.  Trimmed categorical cross‐entropy for deep learning with label noise , 2019, Electronics Letters.

[4]  Moumen T. El-Melegy,et al.  Robust Training of Artificial Feedforward Neural Networks , 2009, Foundations of Computational Intelligence.

[5]  Arash Vahdat,et al.  Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks , 2017, NIPS.

[6]  Abhinav Gupta,et al.  Learning from Noisy Large-Scale Datasets with Minimal Supervision , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ross B. Girshick,et al.  Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Andrzej Rusiecki,et al.  Robust learning algorithm based on LTA estimator , 2013, Neurocomputing.

[9]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[10]  Geoffrey E. Hinton,et al.  Who Said What: Modeling Individual Labelers Improves Classification , 2017, AAAI.

[11]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[12]  Francisco Herrera,et al.  Prototype Selection for Nearest Neighbor Classification: Taxonomy and Empirical Study , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Andrzej Rusiecki.  Robust Learning Algorithm Based on Iterative Least Median of Squares , 2012, Neural Processing Letters.

[14]  Richard Nock,et al.  Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Nagarajan Natarajan,et al.  Learning with Noisy Labels , 2013, NIPS.

[16]  Kadir Liano,et al.  Robust error measure for supervised neural network learning with outliers , 1996, IEEE Trans. Neural Networks.

[17]  Shun-Feng Su,et al.  The annealing robust backpropagation (ARBP) learning algorithm , 2000, IEEE Trans. Neural Networks Learn. Syst..

[18]  Ramesh C. Jain,et al.  A robust backpropagation learning algorithm for function approximation , 1994, IEEE Trans. Neural Networks.

[19]  Dennis L. Wilson,et al.  Asymptotic Properties of Nearest Neighbor Rules Using Edited Data , 1972, IEEE Trans. Syst. Man Cybern..

[20]  Pietro Perona,et al.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Werner A. Stahel,et al.  Robust Statistics: The Approach Based on Influence Functions , 1987 .

[22]  Allan Jabri,et al.  Learning Visual Features from Large Weakly Supervised Data , 2015, ECCV.