A new three-term backpropagation algorithm with convergence analysis

The backpropagation (BP) algorithm is commonly used in many applications, including robotics, automation and weight changes of artificial neural networks (ANNs). This paper proposes the addition of an extra term, a proportional factor (PF), to the standard BP algorithm to speed up the weight adjusting process. The proposed algorithm is tested and the results show that the proposed algorithm outperforms the conventional BP algorithm in convergence speed and the ability to escape from learning stalls. The paper presents a convergence analysis of the three-term BP algorithm. It is shown that if the learning parameters of the three-term BP algorithm satisfies certain conditions given in this paper, then it is guaranteed that the system is stable and will converge to a local minimum. The paper shows that all the local minima of the cost function are stable for the three-term backpropagation algorithm.

[1]  Sagar V. Kamarthi,et al.  Accelerating neural network training using weight extrapolations , 1999, Neural Networks.

[2]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[3]  Marco Gori,et al.  Optimal convergence of on-line backpropagation , 1996, IEEE Trans. Neural Networks.

[4]  James R. Leigh.  Applied Digital Control , 1985 .

[5]  José Carlos Príncipe,et al.  Incremental backpropagation learning networks , 1996, IEEE Trans. Neural Networks.

[6]  J. L. Goldberg.  Matrix theory with applications , 1991 .

[7]  John G. Taylor,et al.  Dynamics of multilayer networks in the vicinity of temporary minima , 1999, Neural Networks.

[8]  Tommy W. S. Chow,et al.  A weight initialization method for improving training speed in feedforward neural network , 2000, Neurocomputing.

[9]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[10]  Okyay Kaynak,et al.  Stabilizing and robustifying the error backpropagation method in neurocontrol applications , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[11]  Yahya H. Zweiri,et al.  A three-term backpropagation algorithm , 2003, Neurocomputing.

[12]  Charles L. Phillips,et al.  Feedback Control Systems , 1988 .