Boundedness and convergence of batch back-propagation algorithm with penalty for feedforward neural networks

This paper investigates the batch back-propagation algorithm with penalty for training feedforward neural networks. A usual penalty is considered, which is a term proportional to the norm of the weights. The learning rate is set to be a small constant or an adaptive series. The main contribution of this paper is to theoretically prove the boundedness of the weights in the network training process. This boundedness is then used to prove some convergence results of the algorithm, which cover both the weak and strong convergence. Simulation results are given to support the theoretical findings.

[1]  Wei Wu,et al.  Convergence of Batch BP Algorithm with Penalty for FNN Training , 2006, ICONIP.

[2]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[3]  J. De Jesús Rubio,et al.  A new discrete-time sliding-mode control with time-varying gain and neural identification , 2006 .

[4]  Wei Wu,et al.  Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks , 2007, Neural Processing Letters.

[5]  Terrence L. Fine,et al.  Parameter Convergence and Learning Curves for Neural Networks , 1999, Neural Computation.

[6]  Gaofeng Zheng,et al.  Boundedness and convergence of online gradient method with penalty and momentum , 2011, Neurocomputing.

[7]  Wen Yu,et al.  Nonlinear system identification using discrete-time recurrent neural networks with stable learning algorithms , 2004, Inf. Sci..

[8]  Wei Wu,et al.  Convergence of approximated gradient method for Elman network , 2008 .

[9]  Wen Yu,et al.  Recurrent Neural Networks Training With Stable Bounding Ellipsoid Algorithm , 2009, IEEE Transactions on Neural Networks.

[10]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[11]  O. Mangasarian,et al.  Serial and parallel backpropagation convergence via nonmonotone perturbed minimization , 1994 .

[12]  Jacek M. Zurada,et al.  Deterministic convergence of conjugate gradient method for feedforward neural networks , 2011, Neurocomputing.

[13]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[14]  Hongmei Shao,et al.  Convergence of an online gradient method with inner-product penalty and adaptive momentum , 2012, Neurocomputing.

[15]  Lijun Liu,et al.  Convergence Analysis of Three Classes of Split-Complex Gradient Algorithms for Complex-Valued Recurrent Neural Networks , 2010, Neural Computation.

[16]  John N. Tsitsiklis,et al.  Gradient Convergence in Gradient methods with Errors , 1999, SIAM J. Optim..

[17]  Mahmood Fathy,et al.  AVLR-EBP: A Variable Step Size Approach to Speed-up the Convergence of Error Back-Propagation Algorithm , 2011, Neural Processing Letters.

[18]  José de Jesús Rubio,et al.  SOFMLS: Online Self-Organizing Fuzzy Modified Least-Squares Network , 2009, IEEE Transactions on Fuzzy Systems.

[19]  Wei Wu,et al.  Deterministic convergence of an online gradient method for BP neural networks , 2005, IEEE Transactions on Neural Networks.

[20]  Alexei A. Gaivoronski,et al.  Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part 1 , 1994 .

[21]  Wei Wu,et al.  Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks , 2009, IEEE Transactions on Neural Networks.

[22]  Plamen P. Angelov,et al.  Uniformly Stable Backpropagation Algorithm to Train a Feedforward Neural Network , 2011, IEEE Transactions on Neural Networks.