A backpropagation algorithm with adaptive learning rate and momentum coefficient

Slower convergence and longer training times are the disadvantages often mentioned when the conventional backpropagation (BP) algorithm are compared with other competing techniques. In addition, in the conventional BP algorithm, the learning rate is fixed and that it is uniform for all the weights in a layer. In this paper, we propose an efficient acceleration technique, the backpropagation with adaptive learning rate and momentum term, which is based on the conventional BP algorithm by employing an adaptive learning rate and momentum factor, where the learning rate and momentum rate are adjusted at each iteration to reduce the training time. Simulation results indicate a superior convergence speed as compared to other competing methods.

[1]  Dan W. Patterson,et al.  Artificial Neural Networks: Theory and Applications , 1998 .

[2]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[3]  Eric B. Baum,et al.  Constructing Hidden Units Using Examples and Queries , 1990, NIPS.

[4]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[5]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[6]  Martin G. Bello,et al.  Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks , 1992, IEEE Trans. Neural Networks.

[7]  Shixin Cheng,et al.  Dynamic learning rate optimization of the backpropagation algorithm , 1995, IEEE Trans. Neural Networks.

[8]  Nazif Tepedelenlioglu,et al.  A fast new algorithm for training feedforward neural networks , 1992, IEEE Trans. Signal Process..

[9]  Anastasios N. Venetsanopoulos,et al.  Fast learning algorithms for neural networks , 1992 .

[10]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[11]  Richard P. Brent,et al.  Fast training algorithms for multilayer neural nets , 1991, IEEE Trans. Neural Networks.

[12]  Tom Tollenaere,et al.  SuperSAB: Fast adaptive back propagation with good scaling properties , 1990, Neural Networks.

[13]  Robert J. Marks,et al.  An adaptively trained neural network , 1991, IEEE Trans. Neural Networks.

[14]  Bernard Widrow,et al.  Sensitivity of feedforward neural networks to weight errors , 1990, IEEE Trans. Neural Networks.

[15]  K. Lang,et al.  Learning to tell two spirals apart , 1988 .