论文信息 - Training recurrent network with block-diagonal approximated Levenberg-Marquardt algorithm

Training recurrent network with block-diagonal approximated Levenberg-Marquardt algorithm

We propose the block-diagonal matrix to approximate the Hessian matrix in the Levenberg-Marquardt method in the training of neural networks. Two weight updating strategies, namely asynchronous and synchronous updating methods, were investigated. Asynchronous method updates weights of one block at a time while synchronous method updates all weights at the same time. Variations of these two methods, which involves the determination of the parameters /spl mu/ and /spl lambda/, are examined.

Lai-Wan Chan | Chi-Cheong Szeto | L. Chan | Chi-Cheong Szeto

[1] Stefanos Kollias,et al. An adaptive least squares algorithm for the efficient training of artificial neural networks , 1989 .

[2] Lars Kai Hansen,et al. Recurrent Networks: Second Order Properties and Pruning , 1994, NIPS.

[3] S. Kollias,et al. Adaptive training of multilayer neural networks using a least squares estimation technique , 1988, IEEE 1988 International Conference on Neural Networks.

[4] J. Wille,et al. On the structure of the Hessian matrix in feedforward networks and second derivative methods , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[5] S. Ragazzini,et al. Learning of word stress in a sub-optimal second order back-propagation neural network , 1988, IEEE 1988 International Conference on Neural Networks.

[6] Yann LeCun,et al. Improving the convergence of back-propagation learning with second-order methods , 1989 .