In gradient descent learning algorithms such as error backpropagation, the learning rate parameter can have a significant effect on generalization accuracy. In particular, decreasing the learning rate below that which yields the fastest convergence can significantly improve generalization accuracy, especially on large, complex problems. The learning rate also directly affects training speed, but not necessarily in the way that many people expect. Many neural network practitioners currently attempt to use the largest learning rate that still allows for convergence, in order to improve training speed. However, a learning rate that is too large can be as slow as a learning rate that is too small, and a learning rate that is too large or too small can require orders of magnitude more training time than one that is in an appropriate range. The paper illustrates how the learning rate affects training speed and generalization accuracy, and thus gives guidelines on how to efficiently select a learning rate that maximizes generalization accuracy.
[1]
Robert A. Jacobs,et al.
Increased rates of convergence through learning rate adaptation
,
1987,
Neural Networks.
[2]
Tom Tollenaere,et al.
SuperSAB: Fast adaptive back propagation with good scaling properties
,
1990,
Neural Networks.
[3]
Philip D. Wasserman,et al.
Advanced methods in neural computing
,
1993,
VNR computer library.
[4]
Heekuck Oh,et al.
Neural Networks for Pattern Recognition
,
1993,
Adv. Comput..
[5]
M.H. Hassoun,et al.
Fundamentals of Artificial Neural Networks
,
1996,
Proceedings of the IEEE.
[6]
Tony R. Martinez,et al.
The Inefficiency of Batch Training for Large Training Sets
,
2000,
IJCNN.