Adaptive Learning-Rate Selection for BPNN Using Evolutionary Programming

The learning speed of a back-propagation neural network greatly depends on its learning rate. For selecting proper learning rates, many approaches-empirical, deterministic, and stochastic methods-have been introduced to date. Some researchers have also tried to find sub-optimal learning rates using various techniques at each training step. This paper proposes a new stochastic method. Our method selects sub-optimal learning rates by an evolutionary adaptation of learning rates for each layer at every training step. We experimented with one relatively simple mapping problem and two complex mapping problems using our method and three typical adaptive methods. Through experiments, we found that the performance of our method is superior to those of the three other methods, especially where the mapping problems are complex. However, evolutionary adaptation with evolutionary programming method generally takes much longer to execute than other adaptive methods. We also experimented with the effect of the performance of our method according to evolutionary adaptation intervals, called windows. This paper describes our selecting algorithm, the other three methods, the window technique, and experimental results.