A NEW DAMPING STRATEGY OF LEVENBERG-MARQUARDT ALGORITHM FOR MULTILAYER PERCEPTRONS

In this paper, a new adjustment to the damping parameter of the Levenberg-Marquardt algorithm is proposed to save training time and to reduce error oscillations. The damping parameter of the Levenberg-Marquardt algorithm switches between a gradient descent method and the Gauss-Newton method. It also afiects training speed and induces error oscillations when a decay rate is flxed. Therefore, our damping strategy decreases the damping parameter with the inner product between weight vectors to make the Levenberg-Marquardt algorithm be- have more like the Gauss-Newton method, and it increases the damping parameter with a diagonally dominant matrix to make the Levenberg-Marquardt algorithm act like a gradient descent method. We tested two simple classiflcations and a handwritten digit recognition for this work. Simulations showed that our method improved training speed and error oscillations were fewer than those of other algo- rithms.

[1]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[2]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.

[3]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[4]  George D. Magoulas,et al.  Globally convergent algorithms with local learning rates , 2002, IEEE Trans. Neural Networks.

[5]  S. Ergezinger,et al.  An accelerated learning algorithm for multilayer perceptrons: optimization layer by layer , 1995, IEEE Trans. Neural Networks.

[6]  C. Charalambous,et al.  Conjugate gradient algorithm for efficient training of artifi-cial neural networks , 1990 .

[7]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[8]  M. Lampton.  Damping-undamping strategies for the Levenberg-Marquardt nonlinear least-squares method , 1997 .

[9]  Benjamin Rodrigues de Menezes,et al.  On-line neural training algorithm with sliding mode control and adaptive learning rate , 2007, Neurocomputing.

[10]  Daniel W. C. Ho,et al.  A new training and pruning algorithm based on node dependence and Jacobian rank deficiency , 2006, Neurocomputing.

[11]  Shixin Cheng,et al.  Dynamic learning rate optimization of the backpropagation algorithm , 1995, IEEE Trans. Neural Networks.

[12]  M. Fukushima,et al.  On the Rate of Convergence of the Levenberg-Marquardt Method , 2001 .

[13]  Francis T.K. Au,et al.  Acceleration of Levenberg-Marquardt training of neural networks with variable decay rate , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[14]  Martin T. Hagan,et al.  Neural network design , 1995 .

[15]  Miguel Pinzolas,et al.  Neighborhood based Levenberg-Marquardt algorithm for neural network training , 2002, IEEE Trans. Neural Networks.

[16]  Sang-Hoon Oh,et al.  A new error function at hidden layers for past training of multilayer perceptrons , 1999, IEEE Trans. Neural Networks.

[17]  Laxmidhar Behera,et al.  On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[18]  Mohammad Bagher Tavakoli,et al.  Modified Levenberg-Marquardt Method for Neural Networks Training , 2007 .

[19]  Laxmidhar Behera,et al.  Corrections to “On Adaptive Learning Rate That Guarantees Convergence in Feedforward Networks” [Sep 06 1116-1125] , 2008, IEEE Transactions on Neural Networks.

[20]  Michael T. Manry,et al.  A neural network training algorithm utilizing multiple sets of linear equations , 1996, Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers.

[21]  A. K. Rigler,et al.  Accelerating the convergence of the back-propagation method , 1988, Biological Cybernetics.

[22]  Jonathan J. Hull,et al.  A Database for Handwritten Text Recognition Research , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[24]  Miguel Pinzolas,et al.  Improvement of the neighborhood based Levenberg-Marquardt algorithm by local adaptation of the learning coefficient , 2005, IEEE Transactions on Neural Networks.

[25]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[26]  Wlodzislaw Duch,et al.  Variable step search algorithm for feedforward networks , 2008, Neurocomputing.

[27]  Ya-Xiang Yuan,et al.  On the Quadratic Convergence of the Levenberg-Marquardt Method without Nonsingularity Assumption , 2005, Computing.

[28]  B. R. Menezes,et al.  Improving generalization of MLPs with sliding mode control and the Levenberg-Marquardt algorithm , 2007, Neurocomputing.

[29]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.