Neural Network Training with Second Order Algorithms

Second order algorithms are very efficient for neural network training because of their fast convergence. In traditional Implementations of second order algorithms [Hagan and Menhaj 1994], Jacobian matrix is calculated and stored, which may cause memory limitation problems when training large-sized patterns. In this paper, the proposed computation is introduced to solve the memory limitation problem in second order algorithms. The proposed method calculates gradient vector and Hessian matrix directly, without Jacobian matrix storage and multiplication. Memory cost for training is significantly reduced by replacing matrix operations with vector operations. At the same time, training speed is also improved due to the memory reduction. The proposed implementation of second order algorithms can be applied to train basically an unlimited number of patterns.

[1]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[2]  Bogdan M. Wilamowski,et al.  Efficient and reliable training of neural networks , 2009, 2009 2nd Conference on Human System Interactions.

[3]  Okyay Kaynak,et al.  Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks , 2008, IEEE Transactions on Industrial Electronics.

[4]  Hao Yu,et al.  Neural Network Learning Without Backpropagation , 2010, IEEE Transactions on Neural Networks.

[5]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[6]  S. Sathiya Keerthi,et al.  Parallel sequential minimal optimization for the training of support vector machines , 2006, IEEE Trans. Neural Networks.

[7]  Bogdan M. Wilamowski,et al.  How Not to Be Frustrated with Neural Networks , 2009 .

[8]  Bogdan M. Wilamowski,et al.  C++ implementation of neural networks trainer , 2009, 2009 International Conference on Intelligent Engineering Systems.

[9]  B.M. Wilamowski,et al.  Neural network architectures and learning algorithms , 2009, IEEE Industrial Electronics Magazine.

[10]  Hao Yu,et al.  Improved Computation for Levenberg–Marquardt Training , 2010, IEEE Transactions on Neural Networks.

[11]  Derong Liu,et al.  Solving the N-bit parity problem using neural networks , 1999, Neural Networks.