Modified inexact Levenberg–Marquardt methods for solving nonlinear least squares problems

AbstractIn the present paper, we propose a modified inexact Levenberg–Marquardt method (LMM) and its global version by virtue of Armijo, Wolfe or Goldstein line-search schemes to solve nonlinear least squares problems (NLSP), especially for the underdetermined case. Under a local error bound condition, we show that a sequence generated by the modified inexact LMM converges to a solution superlinearly and even quadratically for some special parameters, which improves the corresponding results of the classical inexact LMM in Dan et al. (Optim Methods Softw 17:605–626, 2002). Furthermore, the quadratical convergence of the global version of the modified inexact LMM is also established. Finally, preliminary numerical experiments on some medium/large scale underdetermined NLSP show that our proposed algorithm outperforms the classical inexact LMM.

[1]  J. Frédéric Bonnans,et al.  Second-order Sufficiency and Quadratic Growth for Nonisolated Minima , 1995, Math. Oper. Res..

[2]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[3]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[4]  Masao Fukushima,et al.  Convergence Properties of the Inexact Levenberg-Marquardt Method under Local Error Bound Conditions , 2002, Optim. Methods Softw..

[5]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[6]  Zhi-Quan Luo,et al.  Extension of Hoffman's Error Bound to Polynomial Systems , 1994, SIAM J. Optim..

[7]  Guoyin Li.  On the Asymptotically Well Behaved Functions and Global Error Bound for Convex Polynomials , 2010, SIAM J. Optim..

[8]  Åke Björck,et al.  Numerical methods for least square problems , 1996 .

[9]  Margherita Porcelli,et al.  On the convergence of an inexact Gauss–Newton trust-region method for nonlinear least-squares problems with simple bounds , 2013, Optim. Lett..

[10]  Marcin Studniarski,et al.  Weak Sharp Minima: Characterizations and Sufficient Conditions , 1999, SIAM J. Control. Optim..

[11]  Chong Li,et al.  Group Sparse Optimization via lp, q Regularization , 2016, J. Mach. Learn. Res..

[12]  Jong-Shi Pang,et al.  Error bounds in mathematical programming , 1997, Math. Program..

[13]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[14]  Gui-Hua Lin,et al.  Improved convergence results for a modified Levenberg–Marquardt method for nonlinear equations and applications in MPCC , 2016, Optim. Methods Softw..

[15]  Tjalling J. Ypma,et al.  Historical Development of the Newton-Raphson Method , 1995, SIAM Rev..

[16]  Jinyan Fan,et al.  Convergence Properties of a Self-adaptive Levenberg-Marquardt Algorithm Under Local Error Bound Condition , 2006, Comput. Optim. Appl..

[17]  P. Deuflhard,et al.  Affine Invariant Convergence Theorems for Newton’s Method and Extensions to Related Methods , 1979 .

[18]  A. Fischer,et al.  On the inexactness level of robust Levenberg–Marquardt methods , 2010 .

[19]  A. Hoffman.  On approximate solutions of systems of linear inequalities , 1952 .

[20]  Chong Li,et al.  Convergence behavior of Gauss-Newton's method and extensions of the Smale point estimate theory , 2010, J. Complex..

[21]  W. H. Yang.  Error Bounds for Convex Polynomials , 2009, SIAM J. Optim..

[22]  Michael C. Ferris,et al.  A Gauss—Newton method for convex composite optimization , 1995, Math. Program..

[23]  Andreas Fischer,et al.  A Levenberg-Marquardt method with approximate projections , 2014, Comput. Optim. Appl..

[24]  Xiaoqi Yang,et al.  Inexact subgradient methods for quasi-convex optimization problems , 2015, Eur. J. Oper. Res..

[25]  Chong Li,et al.  Approximate Gauss–Newton methods for solving underdetermined nonlinear least squares problems , 2017 .

[26]  Nancy Nichols,et al.  An investigation of incremental 4D‐Var using non‐tangent linear models , 2005 .

[27]  W. Häuβler,et al.  A Kantorovich-type convergence analysis for the Gauss-Newton-Method , 1986 .

[28]  Chong Li,et al.  On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications , 2016, SIAM J. Optim..

[29]  P. Hansen.  Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion , 1987 .

[30]  Jinyan Fan,et al.  On the convergence rate of the inexact Levenberg-Marquardt method , 2011 .

[31]  K. G. Murty,et al.  Convergence of the steepest descent method for minimizing quasiconvex functions , 1996 .

[32]  Zhi-Quan Luo,et al.  Error bounds for analytic systems and their applications , 1994, Math. Program..

[33]  Ya-Xiang Yuan,et al.  Optimization Theory and Methods: Nonlinear Programming , 2010 .

[34]  Benedetta Morini,et al.  A Gauss–Newton method for solving bound-constrained underdetermined nonlinear systems , 2009, Optim. Methods Softw..

[35]  Ya-Xiang Yuan,et al.  On the Quadratic Convergence of the Levenberg-Marquardt Method without Nonsingularity Assumption , 2005, Computing.

[36]  Chong Li,et al.  Linear convergence of CQ algorithms and applications in gene regulatory network inference , 2017 .

[37]  Andreas Fischer,et al.  A Levenberg-Marquardt algorithm for unconstrained multicriteria optimization , 2008, Oper. Res. Lett..

[38]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[39]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[40]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[41]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[42]  M. Fukushima,et al.  On the Rate of Convergence of the Levenberg-Marquardt Method , 2001 .

[43]  Lei Guo,et al.  Solving Mathematical Programs with Equilibrium Constraints , 2015, J. Optim. Theory Appl..

[44]  Guoyin Li,et al.  Global error bounds for piecewise convex polynomials , 2013, Math. Program..

[45]  Chong Li,et al.  On convergence of the Gauss-Newton method for convex composite optimization , 2002, Math. Program..

[46]  Huynh van Ngai,et al.  Error bounds for convex differentiable inequality systems in Banach spaces , 2005, Math. Program..

[47]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[48]  Jinyan Fan,et al.  Inexact Levenberg-Marquardt method for nonlinear equations , 2004 .

[49]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[50]  Yaoliang Yu,et al.  Generalized Conditional Gradient for Sparse Estimation , 2014, J. Mach. Learn. Res..

[51]  Adi Ben-Israel.  A Newton-Raphson method for the solution of systems of equations , 1966 .

[52]  P. Courtier,et al.  A strategy for operational implementation of 4D‐Var, using an incremental approach , 1994 .

[53]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[54]  M. Ferris,et al.  Weak sharp minima in mathematical programming , 1993 .