Training algorithm based on Newton's method with dynamic error control

The use of Newton's method with dynamic error control as a training algorithm for the backpropagation (BP) neural network is considered. Theoretically, it can be proved that Newton's method is convergent in the second-order while the most widely used steepest-descent method is convergent in the first-order. This suggests that Newton's method might be a faster algorithm for the BP network. The updating equations of the two methods are analyzed in detail to extract some important properties with reference to the error surface characteristics. The common benchmark XOR problem is used to compare the performance of the methods.<<ETX>>

[1]  C.-L. Chen,et al.  Improving the training speed of three-layer feedforward neural nets by optimal estimation of the initial weights , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[2]  Esther Levin,et al.  Accelerated Learning in Layered Neural Networks , 1988, Complex Syst..

[3]  Yoshiki Uchikawa,et al.  Learning process of recurrent neural networks , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[4]  Jong Beom Ra,et al.  Weight value initialization for improving training speed in the backpropagation network , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[5]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[6]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[7]  Jianqiang Yi,et al.  Backpropagation based on the logarithmic error function and elimination of local minima , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[8]  Eric B. Baum,et al.  Supervised Learning of Probability Distributions by Neural Networks , 1987, NIPS.