An algorithm for solving sparse Nonlinear Least Squares problems

AbstractWe introduce a new method for solving Nonlinear Least Squares problems when the Jacobian matrix of the system is large and sparse.The main features of the new method are the following:a)The Gauss-Newton equation is “partially” solved at each iteration using a preconditioned Conjugate Gradient algorithm.b)The new point is obtained using a two-dimensional trust region scheme, similar to the one introduced by Bulteau and Vial. We prove global and local convergence results and we present some numerical experiments.ZusammenfassungEine neue Methode zur Lösung nichtlinearer Least-Squares-Probleme bei hochdimensionaler dünnbesetzter Jakobimatrix wird vorgestellt.Die wichtigsten Charakteristika sind:a)Die Gauß-Newton-Gleichung wird „teilweise” bei jeder Iteration gelöst, wobei eine präkonditionierte konjugierte Gradientenmethode verwendet wird.b)Die neue Lösung wird über eine zweidimensionale trust-region Technik erreicht, ähnlich der von Bulteau und Vial vorgeschlagenen Variante. Globale und lokale Konvergenzaussagen werden bewiesen und anhand einiger numerischer Beispiele demonstriert.

[1]  A. George,et al.  A Comparison of Some Methods for Solving Sparse Linear Least-Squares Problems , 1983 .

[2]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[3]  P. Gill,et al.  Sparse Matrix Methods in Optimization , 1984 .

[4]  J. Vial,et al.  A restricted trust region algorithm for unconstrained optimization , 1985 .

[5]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[6]  D. Le.  A fast and robust unconstrained optimization method requiring minimum storage , 1985, Math. Program..

[7]  Gary K. Leaf,et al.  A Preconditioned Conjugate Gradient Method for Solving a Class of Non-Symmetric Linear Systems , 1981 .

[8]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[9]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[10]  Reinhard Menzel.  On solving nonlinear least-squares problems in case of rankdeficient Jacobians , 2005, Computing.

[11]  H. Schwetlick.  Numerische Lösung nichtlinearer Gleichungen , 1978 .

[12]  J. Dennis,et al.  Direct secant updates of matrix factorizations , 1982 .

[13]  Lenhart K. Schubert.  Modification of a quasi-Newton method for nonlinear equations with a sparse Jacobian , 1970 .

[14]  O. Axelsson,et al.  On the rate of convergence of the preconditioned conjugate gradient method , 1986 .

[15]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[16]  O. Axelsson,et al.  On the eigenvalue distribution of a class of preconditioning methods , 1986 .

[17]  Jorge J. Moré,et al.  Recent Developments in Algorithms and Software for Trust Region Methods , 1982, ISMP.

[18]  Thomas F. Coleman,et al.  Large Sparse Numerical Optimization , 1984, Lecture Notes in Computer Science.

[19]  K. L. Hiebert.  An Evaluation of Mathematical Software that Solves Nonlinear Least Squares Problems , 1981, TOMS.

[20]  Philip E. Gill,et al.  Practical optimization , 1981 .

[21]  P. Boggs.  The convergence of the Ben-Israel iteration for nonlinear least squares problems , 1976 .

[22]  J. Dennis.  SOME COMPUTATIONAL TECHNIQUES FOR THE NONLINEAR LEAST SQUARES PROBLEM , 1973 .

[23]  P. Deuflhard,et al.  A Study of the Gauss-Newton Method for the Solution of Nonlinear Least Squares Problems , 1980 .

[24]  M. Martínez,et al.  Parallel and sequential Kaczmarz methods for solving underdetermined nonlinear equations , 1986 .

[25]  R. T. Cutler,et al.  Tomographic determination of velocity and depth in laterally varying media , 1985 .

[26]  Lourenildo W. B. Leite,et al.  Ridge regression applied to the inversion of two-dimensional aeromagnetic anomalies , 1985 .

[27]  José Mario Martínez,et al.  A Quasi-Newton method with modification of one column per iteration , 1984, Computing.

[28]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[29]  M. A. Ajiz,et al.  A robust incomplete Choleski‐conjugate gradient algorithm , 1984 .

[30]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[31]  Larry Nazareth,et al.  Some Recent Approaches to Solving Large Residual Nonlinear Least Squares Problems , 1980 .

[32]  M. Heath.  Numerical Methods for Large Sparse Linear Least Squares Problems , 1984 .

[33]  L. Pedersen,et al.  Interpretation of Potential Field Data a Generalized Inverse APPROACH , 1977 .

[34]  A. George,et al.  Solution of sparse linear least squares problems using givens rotations , 1980 .

[35]  R. Schaback.  Convergence analysis of the general Gauss-Newton algorithm , 1985 .

[36]  S. Nash.  Truncated-Newton methods , 1982 .

[37]  David F. Shanno.  Globally convergent conjugate gradient algorithms , 1985, Math. Program..

[38]  K. Meyn.  Solution of underdetermined nonlinear equations by stationary iteration methods , 1983 .

[39]  R. Fletcher.  A modified Marquardt subroutine for non-linear least squares , 1971 .

[40]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[41]  O. Axelsson,et al.  On some versions of incomplete block-matrix factorization iterative methods , 1984 .

[42]  J H Martinez,et al.  Solving systems of nonlinear equations by means of an accelerated successive orthogonal projections method , 1986 .

[43]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[44]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[45]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[46]  O. Axelsson.  Incomplete block matrix factorization preconditioning methods. The ultimate answer , 1985 .

[47]  Trond Steihaug,et al.  Truncated-newtono algorithms for large-scale unconstrained optimization , 1983, Math. Program..