Deterministic convergence of chaos injection-based gradient method for training feedforward neural networks

It has been shown that, by adding a chaotic sequence to the weight update during the training of neural networks, the chaos injection-based gradient method (CIBGM) is superior to the standard backpropagation algorithm. This paper presents the theoretical convergence analysis of CIBGM for training feedforward neural networks. We consider both the case of batch learning as well as the case of online learning. Under mild conditions, we prove the weak convergence, i.e., the training error tends to a constant and the gradient of the error function tends to zero. Moreover, the strong convergence of CIBGM is also obtained with the help of an extra condition. The theoretical results are substantiated by a simulation example.

[1]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[2]  Andrew Chi-Sing Leung,et al.  Convergence Analyses on On-Line Weight Noise Injection-Based Training Algorithms for MLPs , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Wei Wu,et al.  Deterministic convergence of an online gradient method for BP neural networks , 2005, IEEE Transactions on Neural Networks.

[4]  Andrew Chi-Sing Leung,et al.  Convergence and Objective Functions of Some Fault/Noise-Injection-Based Online Learning Algorithms for RBF Networks , 2010, IEEE Transactions on Neural Networks.

[5]  Hideaki Sakai,et al.  A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter , 1992, IEEE Trans. Signal Process..

[6]  Stanislaw Osowski,et al.  Fast Second Order Learning Algorithm for Feedforward Multilayer Neural Networks and its Applications , 1996, Neural Networks.

[7]  Terrence L. Fine,et al.  Parameter Convergence and Learning Curves for Neural Networks , 1999, Neural Computation.

[8]  C. Charalambous,et al.  Conjugate gradient algorithm for efficient training of artifi-cial neural networks , 1990 .

[9]  Xin Yu,et al.  Convergence of gradient method with penalty for Ridge Polynomial neural network , 2012, Neurocomputing.

[10]  Gaofeng Zheng,et al.  Boundedness and convergence of online gradient method with penalty and momentum , 2011, Neurocomputing.

[11]  Wei Wu,et al.  Convergence Analysis of Batch Gradient Algorithm for Three Classes of Sigma-Pi Neural Networks , 2007, Neural Processing Letters.

[12]  Wei Wu,et al.  Convergence analysis of online gradient method for BP neural networks , 2011, Neural Networks.

[13]  Daqing Guo,et al.  Inhibition of rhythmic spiking by colored noise in neural systems , 2011, Cognitive Neurodynamics.

[14]  Mengmeng Du,et al.  Noise-induced spatiotemporal patterns in Hodgkin–Huxley neuronal network , 2013, Cognitive Neurodynamics.

[15]  C. Zhang,et al.  Convergence of BP algorithm for product unit neural networks with exponential weights , 2008, Neurocomputing.

[16]  Wei Wu,et al.  Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks , 2009, IEEE Transactions on Neural Networks.

[17]  Shigetoshi Nara,et al.  Novel tracking function of moving target using chaotic dynamics in a recurrent neural network model , 2008, Cognitive Neurodynamics.

[18]  Wei Wu,et al.  Boundedness and convergence of batch back-propagation algorithm with penalty for feedforward neural networks , 2012, Neurocomputing.

[19]  Yoshifumi Nishio,et al.  PERFORMANCE OF CHAOS AND BURST NOISES INJECTED TO THE HOPFIELD NN FOR QUADRATIC ASSIGNMENT PROBLEMS , 2004 .

[20]  Andrew Chi-Sing Leung,et al.  On-Line Node Fault Injection Training Algorithm for MLP Networks: Objective Function and Convergence Analysis , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[22]  Ehud D. Karnin,et al.  A simple procedure for pruning back-propagation trained neural networks , 1990, IEEE Trans. Neural Networks.

[23]  Xiaodong Liu,et al.  Convergence analysis of fully complex backpropagation algorithm based on Wirtinger calculus , 2013, Cognitive Neurodynamics.

[24]  Jacek M. Zurada,et al.  Convergence of online gradient method for feedforward neural networks with smoothing L1/2 regularization penalty , 2014, Neurocomputing.

[25]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[26]  Marius-F. Danca,et al.  Noise induced complexity: patterns and collective phenomena in a small-world neuronal network , 2013, Cognitive Neurodynamics.

[27]  Shigetoshi Nara,et al.  Chaotic neural network applied to two-dimensional motion control , 2010, Cognitive Neurodynamics.

[28]  Kazuyuki Murase,et al.  Injecting Chaos in Feedforward Neural Networks , 2011, Neural Processing Letters.

[29]  Tom Heskes,et al.  A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning , 1996, IEEE Trans. Neural Networks.

[30]  Wei Wu,et al.  Convergence of gradient method with momentum for two-Layer feedforward neural networks , 2006, IEEE Transactions on Neural Networks.

[31]  Jacek M. Zurada,et al.  Deterministic convergence of conjugate gradient method for feedforward neural networks , 2011, Neurocomputing.

[32]  John N. Tsitsiklis,et al.  Gradient Convergence in Gradient methods with Errors , 1999, SIAM J. Optim..