Improved Computation for Levenberg–Marquardt Training

The improved computation presented in this paper is aimed to optimize the neural networks learning process using Levenberg-Marquardt (LM) algorithm. Quasi-Hessian matrix and gradient vector are computed directly, without Jacobian matrix multiplication and storage. The memory limitation problem for LM training is solved. Considering the symmetry of quasi-Hessian matrix, only elements in its upper/lower triangular array need to be calculated. Therefore, training speed is improved significantly, not only because of the smaller array stored in memory, but also the reduced operations in quasi-Hessian matrix calculation. The improved memory and time efficiencies are especially true for large sized patterns training.

[1]  Ju-Jang Lee,et al.  Training Two-Layered Feedforward Networks With Variable Projection Method , 2008, IEEE Transactions on Neural Networks.

[2]  V. Fernão Pires,et al.  Unsupervised Neural-Network-Based Algorithm for an On-Line Diagnosis of Three-Phase Induction Motor Stator Fault , 2007, IEEE Transactions on Industrial Electronics.

[3]  George W. Irwin,et al.  A New Jacobian Matrix for Optimal Learning of Single-Layer Neural Networks , 2008, IEEE Transactions on Neural Networks.

[4]  José R. Álvarez.  Injecting Knowledge into the Solution of the Two-Spiral Problem , 1999, Neural Computing & Applications.

[5]  Okyay Kaynak,et al.  Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks , 2008, IEEE Transactions on Industrial Electronics.

[6]  Zhen Zhu,et al.  Optimized Approximation Algorithm in Neural Networks Without Overfitting , 2008, IEEE Transactions on Neural Networks.

[7]  Edgar N. Sánchez,et al.  Adaptive Approximation Based Control: Unifying Neural, Fuzzy and Traditional Adaptive Approximation Approaches (Farrell, J.A. and Polycarpou, M.M. [Book review] , 2008, IEEE Trans. Neural Networks.

[8]  L.M. Tolbert,et al.  Fault Diagnostic System for a Multilevel Inverter Using a Neural Network , 2007, IEEE Transactions on Power Electronics.

[9]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[10]  Bogdan M. Wilamowski,et al.  Efficient and reliable training of neural networks , 2009, 2009 2nd Conference on Human System Interactions.

[11]  P. S. Sastry,et al.  Analysis of the back-propagation algorithm with momentum , 1994, IEEE Trans. Neural Networks.

[12]  Derong Liu,et al.  Adaptive approximation based control: Unifying neural, fuzzy and traditional adaptive approximation approaches. Jay A. Farrell and Marios M. Polycarpou, Wiley, New York, 2006. No of pages: 440. ISBN 978-0-471-72788-0 , 2008 .

[13]  Anastasios Tefas,et al.  Weighted Piecewise LDA for Solving the Small Sample Size Problem in Face Verification , 2007, IEEE Transactions on Neural Networks.

[14]  Audra E. Kosh,et al.  Linear Algebra and its Applications , 1992 .

[15]  P. J. Werbos,et al.  Backpropagation: past and future , 1988, IEEE 1988 International Conference on Neural Networks.

[16]  Jiann-Ming Wu,et al.  Multilayer Potts Perceptrons With Levenberg–Marquardt Learning , 2008, IEEE Transactions on Neural Networks.

[17]  Stavros J. Perantonis,et al.  Two highly efficient second-order algorithms for training feedforward networks , 2002, IEEE Trans. Neural Networks.

[18]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[19]  B.M. Wilamowski,et al.  Neural network architectures and learning algorithms , 2009, IEEE Industrial Electronics Magazine.

[20]  Silvia Ferrari,et al.  A Constrained Optimization Approach to Preserving Prior Knowledge During Incremental Training , 2008, IEEE Transactions on Neural Networks.

[21]  Boaz Lerner,et al.  Accurate and Fast Off and Online Fuzzy ARTMAP-Based Image Classification With Application to Genetic Abnormality Diagnosis , 2006, IEEE Transactions on Neural Networks.

[22]  Alexander G. Loukianov,et al.  Discrete-Time Adaptive Backstepping Nonlinear Control via High-Order Neural Networks , 2007, IEEE Transactions on Neural Networks.

[23]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[24]  S. Sathiya Keerthi,et al.  Parallel sequential minimal optimization for the training of support vector machines , 2006, IEEE Trans. Neural Networks.

[25]  Bogdan M. Wilamowski,et al.  How Not to Be Frustrated with Neural Networks , 2009 .