论文信息 - Automatic Adaptation of Learning Rate for Backpropagation Neural Networks

Automatic Adaptation of Learning Rate for Backpropagation Neural Networks

A method improving the convergence rate of the backpropagation algorithm is proposed. This method adapts the learning rate using the Barzilai and Borwein [IMA J.Numer. Anal., 8, 141– 148, 1988] steplength update for gradient descent methods. The determined learning rate is different for each epoch and depends on the weights and gradient values of the previous one. Experimental results show that the proposed method considerably improves the convergence rates of the backpropagation algorithm and, for the chosen test problems, outperforms other well-known training methods.

Vassilis P. Plagianakos | Michael N. Vrahatis | D. G. Sotiropoulos | V. Plagianakos | M. Vrahatis

[1] Elijah Polak,et al. Optimization: Algorithms and Consistent Approximations , 1997 .

[2] Scott E. Fahlman,et al. An empirical study of learning speed in back-propagation networks , 1988 .

[3] Robert A. Jacobs,et al. Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[4] D. E. Rumelhart,et al. chapter Parallel Distributed Processing, Exploration in the Microstructure of Cognition , 1986 .

[5] J. Borwein,et al. Two-Point Step Size Gradient Methods , 1988 .

[6] Bernard Widrow,et al. Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[7] G. Kane. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .