Robust Multilayer Perceptrons: Robust Loss Functions and Their Derivatives

Common types of artificial neural networks have been well known to suffer from the presence of outlying measurements (outliers) in the data. However, there are only a few available robust alternatives for training common form of neural networks. In this work, we investigate robust fitting of multilayer perceptrons, i.e. alternative approaches to the most common type of feedforward neural networks. Particularly, we consider robust neural networks based on the robust loss function of the least trimmed squares, for which we express formulas for derivatives of the loss functions. Some formulas, which are however incorrect, have been already available. Further, we consider a very recently proposed multilayer perceptron based on the loss function of the least weighted squares, which appears a promising highly robust approach. We also derive the derivatives of the loss functions, which are to the best of our knowledge a novel contribution of this paper. The derivatives may find applications in implementations of the robust neural networks, if a (gradient-based) backpropagation algorithm is used.

[1]  Jan Ámos Vísek,et al.  Consistency of the least weighted squares under heteroscedasticity , 2011, Kybernetika.

[2]  Petra Vidnerová,et al.  On Robust Training of Regression Neural Networks , 2020 .

[3]  Andrzej Rusiecki,et al.  Training Neural Networks on Noisy Data , 2014, ICAISC.

[4]  PETER J. ROUSSEEUW,et al.  Computing LTS Regression for Large Data Sets , 2005, Data Mining and Knowledge Discovery.

[5]  Andrzej Rusiecki,et al.  Robust learning algorithm based on LTA estimator , 2013, Neurocomputing.

[6]  Jan Ámos Vísek,et al.  The least trimmed squares. Part I: Consistency , 2006, Kybernetika.

[7]  Jan Kalina,et al.  Robust Training of Radial Basis Function Neural Networks , 2019, ICAISC.

[8]  Pierpaolo D'Urso,et al.  Fuzzy c-ordered medoids clustering for interval-valued data , 2016, Pattern Recognit..

[9]  Jan Kalina,et al.  A Robust Supervised Variable Selection for Noisy High-Dimensional Data , 2015, BioMed research international.

[10]  A. Ullah,et al.  The Oxford handbook of applied nonparametric and semiparametric econometrics and statistics , 2014 .

[11]  Andrzej Rusiecki,et al.  Reducing noise impact on MLP training , 2016, Soft Comput..

[12]  J. Yearwood,et al.  Derivative-free optimization and neural networks for robust regression , 2012 .

[13]  KordosMirosław,et al.  Reducing noise impact on MLP training , 2016 .

[14]  Jan Kalina.  A robust pre-processing of BeadChip microarray images , 2018 .

[15]  Jesús Armengol,et al.  Myopia Control with a Novel Peripheral Gradient Soft Lens and Orthokeratology: A 2-Year Clinical Trial , 2015, BioMed research international.

[16]  Erol Egrioglu,et al.  Robust multilayer neural network based on median neuron model , 2012, Neural Computing and Applications.

[17]  Heiko Paulheim,et al.  A decomposition of the outlier detection problem into a set of supervised learning problems , 2015, Machine Learning.

[18]  Moustapha Cissé,et al.  Parseval Networks: Improving Robustness to Adversarial Examples , 2017, ICML.

[19]  Christopher F. Parmeter,et al.  Applied Nonparametric Econometrics , 2015 .