A structured diagonal Hessian approximation method with evaluation complexity analysis for nonlinear least squares

This work proposes a Jacobian-free strategy for addressing large-scale nonlinear least-squares problems, in which structured secant conditions are used to define a diagonal approximation for the Hessian matrix. Proper safeguards are devised to ensure descent directions along the generated sequence. Worst-case evaluation analysis is provided within the framework of a non-monotone line search. Numerical experiments contextualize the proposed strategy, by addressing structured problems from the literature, also solved by related and recently presented conjugate gradient and multivariate spectral gradient strategies, as well as the classic Fletcher–Reeves conjugate gradient, and the Raydan–Barzilai–Borwein methods. The comparative computational results show a favorable performance of the proposed approach, mainly as far as robustness is concerned.

[1]  Wei Xu,et al.  A secant method for nonlinear least-squares minimization , 2012, Comput. Optim. Appl..

[2]  Yurii Nesterov,et al.  Modified Gauss–Newton scheme with worst case guarantees for global performance , 2007, Optim. Methods Softw..

[3]  Feng Ding,et al.  Maximum likelihood least squares identification method for input nonlinear finite impulse response moving average systems , 2012, Math. Comput. Model..

[4]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[5]  Hiroshi Yabe,et al.  Nonlinear conjugate gradient methods with structured secant condition for nonlinear least squares problems , 2010, J. Comput. Appl. Math..

[6]  John E. Dennis,et al.  A New Algorithm for Nonlinear Least Squares Curve Fitting , 1970 .

[7]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Constrained Nonlinear Least-Squares and General Constrained Nonlinear Optimization Using Second-Order Methods , 2015, SIAM J. Numer. Anal..

[8]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[9]  J. Betts.  Solving the nonlinear least square problem: Application of a general method , 1976 .

[10]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[11]  Y. Nesterov.  Modified Gauss-Newton Scheme with Worst-Case Guarantees for its Global Performance , 2003 .

[12]  Li Min Tang,et al.  A Regularization Homotopy Iterative Method for Ill-Posed Nonlinear Least Squares Problem and its Application , 2011 .

[13]  M. Raydan.  On the Barzilai and Borwein choice of steplength for the gradient method , 1993 .

[14]  M. J. D. Powell,et al.  A Method for Minimizing a Sum of Squares of Non-Linear Functions Without Calculating Derivatives , 1965, Comput. J..

[15]  Jinyan Fan,et al.  Global complexity bound of the Levenberg–Marquardt method , 2016, Optim. Methods Softw..

[16]  J. J. McKeown,et al.  Specialised versus general-purpose algorithms for minimising functions that are sums of squared terms , 1975, Math. Program..

[17]  M. Al-Baali.  Quasi-Newton Algorithms for Large-Scale Nonlinear Least-Squares , 2003 .

[18]  Jorge J. Moré,et al.  Benchmarking optimization software with performance profiles , 2001, Math. Program..

[19]  Marcos Raydan,et al.  The Barzilai and Borwein Gradient Method for the Large Scale Unconstrained Minimization Problem , 1997, SIAM J. Optim..

[20]  José Mario Martínez,et al.  Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models , 2017, Math. Program..

[21]  Maria Teresa Vespucci,et al.  Numerical experiments with variations of the Gauss-Newton algorithm for nonlinear least squares , 1988 .

[22]  Anastasia Cornelio,et al.  Regularized nonlinear least squares methods for hit position reconstruction in small gamma cameras , 2011, Appl. Math. Comput..

[23]  Wei Xu,et al.  Jacobian-Free Implicit Inner-Iteration Preconditioner for Nonlinear Least Squares Problems , 2016, J. Sci. Comput..

[24]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[25]  William W. Hager,et al.  A Nonmonotone Line Search Technique and Its Application to Unconstrained Optimization , 2004, SIAM J. Optim..

[26]  Xin-She Yang,et al.  A literature survey of benchmark functions for global optimisation problems , 2013, Int. J. Math. Model. Numer. Optimisation.

[27]  Sandra A. Santos,et al.  An inexact and nonmonotone proximal method for smooth unconstrained minimization , 2014, J. Comput. Appl. Math..

[28]  J. Dennis.  SOME COMPUTATIONAL TECHNIQUES FOR THE NONLINEAR LEAST SQUARES PROBLEM , 1973 .

[29]  Åke Björck,et al.  Numerical methods for least square problems , 1996 .

[30]  Ya-Xiang Yuan,et al.  Optimization Theory and Methods: Nonlinear Programming , 2010 .

[31]  Jürgen Huschens,et al.  On the Use of Product Structure in Secant Methods for Nonlinear Least Squares Problems , 1994, SIAM J. Optim..

[32]  L. Luksan.  Hybrid methods for large sparse nonlinear least squares , 1996 .

[33]  Katya Scheinberg,et al.  On the local convergence of a derivative-free algorithm for least-squares minimization , 2010, Computational Optimization and Applications.

[34]  Mike C. Bartholomew-Biggs,et al.  The estimation of the hessian matrix in nonlinear least squares problems with non-zero residuals , 1977, Math. Program..

[35]  Larry Nazareth,et al.  Some Recent Approaches to Solving Large Residual Nonlinear Least Squares Problems , 1980 .

[36]  H. Hartley.  The Modified Gauss-Newton Method for the Fitting of Non-Linear Regression Functions by Least Squares , 1961 .

[37]  Stefan Henn.  A Levenberg–Marquardt Scheme for Nonlinear Image Registration , 2003 .

[38]  Hans Bruun Nielsen.  UCTP - Test Problems for Unconstrained Optimization , 2000 .

[39]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[40]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[41]  D. D. Morrison.  Methods for nonlinear least squares problems and convergence proofs , 1960 .

[42]  Günter Wozny,et al.  Nonlinear ill-posed problem analysis in model-based parameter estimation and experimental design , 2015, Comput. Chem. Eng..

[43]  D. Keyes,et al.  Jacobian-free Newton-Krylov methods: a survey of approaches and applications , 2004 .

[44]  Nicholas I. M. Gould,et al.  Convergence of a Regularized Euclidean Residual Algorithm for Nonlinear Least-Squares , 2010, SIAM J. Numer. Anal..

[45]  L. Qi,et al.  Global Convergence of Gauss-Newton-MBFGS Method for Solving the Nonlinear Least Squares Problem , 2010 .

[46]  R. Fletcher,et al.  Hybrid Methods for Nonlinear Least Squares , 1987 .

[47]  G. Golub,et al.  Separable nonlinear least squares: the variable projection method and its applications , 2003 .

[48]  Douglas Soares Gonçalves,et al.  Local analysis of a spectral correction for the Gauss-Newton model applied to quadratic residual problems , 2016, Numerical Algorithms.

[49]  Le Han,et al.  Multivariate spectral gradient method for unconstrained optimization , 2008, Appl. Math. Comput..

[50]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[51]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[52]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[53]  Ya-xiang Yuan,et al.  On the convergence and worst-case complexity of trust-region and regularization methods for unconstrained optimization , 2015, Math. Program..

[54]  Nicholas I. M. Gould,et al.  On the Evaluation Complexity of Cubic Regularization Methods for Potentially Rank-Deficient Nonlinear Least-Squares Problems and Its Relevance to Constrained Nonlinear Optimization , 2013, SIAM J. Optim..

[55]  L. Grippo,et al.  A nonmonotone line search technique for Newton's method , 1986 .

[56]  M. Al-Baali,et al.  Variational Methods for Non-Linear Least-Squares , 1985 .