A New BP Algorithm with Adaptive Momentum for FNNs Training

In this paper, a new back propagation (BP) algorithm with adaptive momentum is proposed, where the momentum coefficient is adjusted iteratively based on the current descent direction and the weight increment in the last iteration. A convergence result of the algorithm is presented when it is used for training feed forward neural networks (FNNs) with a hidden layer. Simulation results have shown that this new algorithm has a distinct superiority in fast convergence and smoothing oscillation over the conventional BP method. Moreover, the range for the learning rate has been widened after the inclusion of such an adaptable momentum while maintaining the stability of networks.

[1]  John Mark Bishop,et al.  Simple adaptive momentum: New algorithm for training multilayer perceptrons , 1994 .

[2]  T. J. Terrell,et al.  Accelerated training of backpropagation networks by using adaptive momentum step , 1992 .

[3]  Bin-Da Liu,et al.  A backpropagation algorithm with adaptive learning rate and momentum coefficient , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[4]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[5]  Tony R. Martinez,et al.  The need for small learning rates on large problems , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[6]  Zhang Suzhen,et al.  Variable-step BP training algorithm with an adaptive momentum term , 2002, Proceedings of the 4th World Congress on Intelligent Control and Automation (Cat. No.02EX527).

[7]  Tony R. Martinez,et al.  Improved Backpropagation Learning in Neural Networks with Windowed Momentum , 2002, Int. J. Neural Syst..

[8]  W. C. Miller,et al.  A new acceleration technique for the backpropagation algorithm , 1993, IEEE International Conference on Neural Networks.

[9]  Martin T. Hagan,et al.  Neural network design , 1995 .

[10]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[11]  J. Nazuno.  Haykin, Simon. Neural networks: A comprehensive foundation, Prentice Hall, Inc. Segunda Edición, 1999 , 2000 .

[12]  Frank Fallside,et al.  An adaptive training algorithm for back propagation networks , 1987 .

[13]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .