MCP Based Noise Resistant Algorithm for Training RBF Networks and Selecting Centers

In the implementation of a neural network, some imperfect issues, such as precision error and thermal noise, always exist. They can be modeled as multiplicative noise. This paper studies the problem of training RBF network and selecting centers under multiplicative noise. We devise a noise resistant training algorithm based on the alternating direction method of multipliers (ADMM) framework and the minimax concave penalty (MCP) function. Our algorithm first uses all training samples to create the RBF nodes. Afterwards, we derive the training objective function that can tolerate to the existence of noise. Finally, we add a MCP term to the objective function. We then apply the ADMM framework to minimize the modified objective function. During training, the MCP term has an ability to make some unimportant RBF weights to zero. Hence training and RBF node selection can be done at the same time. The proposed algorithm is called the ADMM-MCP algorithm. Also, we present the convergent properties of the ADMM-MCP algorithm. From the simulation result, the ADMM-MCP algorithm is better than many other RBF training algorithms under weight/node noise situation.

[1]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[2]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[3]  Andrew Chi-Sing Leung,et al.  A Regularizer Approach for RBF Networks Under the Concurrent Weight Failure Situation , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[5]  Wotao Yin,et al.  Global Convergence of ADMM in Nonconvex Nonsmooth Optimization , 2015, Journal of Scientific Computing.

[6]  Dmitry M. Malioutov,et al.  Homotopy continuation for sparse signal representation , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[7]  E.E. Swartzlander,et al.  Digital neural network implementation , 1992, Eleventh Annual International Phoenix Conference on Computers and Communication [1992 Conference Proceedings].

[8]  Jian Huang,et al.  COORDINATE DESCENT ALGORITHMS FOR NONCONVEX PENALIZED REGRESSION, WITH APPLICATIONS TO BIOLOGICAL FEATURE SELECTION. , 2011, The annals of applied statistics.

[9]  Xiaolin Hu,et al.  Comparison of $\ell _{1}$ -Norm SVR and Sparse Coding Algorithms for Linear Regression , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[10]  J. Nazuno.  Haykin, Simon. Neural networks: A comprehensive foundation, Prentice Hall, Inc. Segunda Edición, 1999 , 2000 .

[11]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[12]  Andrew Chi-Sing Leung,et al.  Online training and its convergence for faulty networks with multiplicative weight noise , 2015, Neurocomputing.

[13]  Ignacio Rojas,et al.  A Quantitative Study of Fault Tolerance, Noise Immunity, and Generalization Ability of MLPs , 2000, Neural Computation.

[14]  Dingli Yu,et al.  Selecting radial basis function network centers with recursive orthogonal least squares training , 2000, IEEE Trans. Neural Networks Learn. Syst..

[15]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .