Extended Levenberg-Marquardt Method for Composite Function Minimization

In this paper, we propose an extended Levenberg-Marquardt (ELM) framework that generalizes the classic Levenberg-Marquardt (LM) method to solve the unconstrained minimization problem min ρ ( r ( x )), where r : R n → R m and ρ : R m → R . We also develop a few inexact variants which generalize ELM to the cases where the inner subproblem is not solved exactly and the Jacobian is simplified, or perturbed. Global convergence and local superlinear convergence are established under certain suitable conditions. Numerical results show that our methods are promising. 90C30, 90C53.

[1]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[2]  Jinyan Fan.  A Shamanskii-like Levenberg-Marquardt method for nonlinear equations , 2013, Comput. Optim. Appl..

[3]  P. Toint,et al.  CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization , 2013, Computational Optimization and Applications.

[4]  M. Friedlander,et al.  Robust inversion, dimensionality reduction, and randomized sampling , 2012, Math. Program..

[5]  Chih-Jen Lin,et al.  Trust Region Newton Method for Logistic Regression , 2008, J. Mach. Learn. Res..

[6]  Ya-Xiang Yuan,et al.  On the Quadratic Convergence of the Levenberg-Marquardt Method without Nonsingularity Assumption , 2005, Computing.

[7]  Jinyan Fan,et al.  Inexact Levenberg-Marquardt method for nonlinear equations , 2004 .

[8]  Masao Fukushima,et al.  Convergence Properties of the Inexact Levenberg-Marquardt Method under Local Error Bound Conditions , 2002, Optim. Methods Softw..

[9]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[10]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[11]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[12]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[13]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[14]  Jinyan Fan,et al.  The modified Levenberg-Marquardt method for nonlinear equations with cubic convergence , 2012, Math. Comput..

[15]  Jinyan Fan,et al.  On the convergence rate of the inexact Levenberg-Marquardt method , 2011 .

[16]  T. Minka.  A comparison of numerical optimizers for logistic regression , 2004 .

[17]  Jin-yanFan.  A MODIFIED LEVENBERG-MARQUARDT ALGORITHM FOR SINGULAR SYSTEM OF NONLINEAR EQUATIONS , 2003 .

[18]  J. Fan,et al.  The convergence of a new Levenberg-Marquardt method , 2001 .

[19]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[20]  M. Fukushima,et al.  On the Rate of Convergence of the Levenberg-Marquardt Method , 2001 .

[21]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .

[22]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .