Algorithms for Nonlinear Least-Squares Problems

Abstract : This paper addresses the nonlinear least-squares problem which arises most often in data fitting applications. Much research has focused on the development of specialized algorithms that attempt to exploit the structure of the nonlinear least-squares objective. The author surveys numerical methods developed for problems in which sparsity in the derivatives of f is not taken into account in formulating algorithms. Keywords: Multivariate functions; Gauss- Newton methods; Levenberg Marquardt methods; Quasi-Newton methods; Quadratic programming; Unconstrained optimization methods. (KR)

[1]  Jorge J. Moré,et al.  Recent Developments in Algorithms and Software for Trust Region Methods , 1982, ISMP.

[2]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[3]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[4]  D. Shanno.  Parameter Selection for Modified Newton Methods for Function Minimization , 1970 .

[5]  W. Gander.  Least squares with a quadratic constraint , 1980 .

[6]  W. Davidon.  New least-square algorithms , 1976 .

[7]  A. Jones,et al.  Spiral - A New Algorithm for Non-Linear Parameter Estimation Using Least Squares , 1970, Comput. J..

[8]  Douglas E. Salane.  A continuation approach for solving large-residual nonlinear least squares problems , 1987 .

[9]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[10]  R. Schaback.  Convergence analysis of the general Gauss-Newton algorithm , 1985 .

[11]  Mike C. Bartholomew-Biggs,et al.  The estimation of the hessian matrix in nonlinear least squares problems with non-zero residuals , 1977, Math. Program..

[12]  Larry Nazareth,et al.  Some Recent Approaches to Solving Large Residual Nonlinear Least Squares Problems , 1980 .

[13]  C. Fraley.  Solution of nonlinear least-squares problems , 1987 .

[14]  P. Deuflhard,et al.  A Study of the Gauss-Newton Method for the Solution of Nonlinear Least Squares Problems , 1980 .

[15]  P. Gill,et al.  Fortran package for constrained linear least-squares and convex quadratic programming. User's Guide for LSSOL (Version 1. 0) , 1986 .

[16]  M. Al-Baali,et al.  Variational Methods for Non-Linear Least-Squares , 1985 .

[17]  M GayDavid,et al.  Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust-Region Approach , 1983 .

[18]  L. W. Cornwell,et al.  Computational experience with Davidon's least-square algorithm , 1980 .

[19]  John E. Dennis,et al.  Algorithm 573: NL2SOL—An Adaptive Nonlinear Least-Squares Algorithm [E4] , 1981, TOMS.

[20]  Jr. John E. Dennis.  A Brief Survey of Convergence Results for Quasi-Newton Methods , 1975 .

[21]  H. Walker,et al.  Convergence Theorems for Least-Change Secant Update Methods, , 1981 .

[22]  Albrecht Preusser.  Remark on algorithm 526 , 1985, TOMS.

[23]  Philip E. Gill,et al.  Practical optimization , 1981 .

[24]  J. Varah.  A Practical Examination of Some Numerical Methods for Linear Discrete Ill-Posed Problems , 1979 .

[25]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[26]  P. Gill,et al.  Model Building and Practical Aspects of Nonlinear Programming , 1985 .

[27]  M. J. D. Powell,et al.  Variable Metric Methods for Constrained Optimization , 1982, ISMP.

[28]  Gene H. Golub,et al.  Matrix computations , 1983 .

[29]  P. Gill,et al.  Nonlinear least squares and nonlinearly constrained optimization , 1976 .

[30]  S. Goldfeld,et al.  Maximization by Quadratic Hill-Climbing , 1966 .

[31]  Christina Fraley.  Software Performance on Nonlinear Least-Squares Problems , 1989 .

[32]  J. Dennis,et al.  Techniques for nonlinear least squares and robust regression , 1978 .

[33]  Michael A. Saunders,et al.  Software and its relationship to methods , 1984 .

[34]  George D. Byrne,et al.  THE PROBLEM OF MINIMIZING NONLINEAR FUNCTIONALS: I. LEAST SQUARES , 1973 .

[35]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[36]  R. Fletcher,et al.  An efficient line search for nonlinear least squares , 1986 .

[37]  G. Golub,et al.  Numerical techniques in mathematical programming , 1970 .

[38]  C. Reinsch.  Smoothing by spline functions , 1967 .

[39]  Yonathan Bard,et al.  Comparison of Gradient Methods for the Solution of Nonlinear Parameter Estimation Problems , 1970 .

[40]  J. Dennis.  SOME COMPUTATIONAL TECHNIQUES FOR THE NONLINEAR LEAST SQUARES PROBLEM , 1973 .

[41]  J. J. McKeown,et al.  Specialised versus general-purpose algorithms for minimising functions that are sums of squared terms , 1975, Math. Program..

[42]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[43]  J. Betts.  Solving the nonlinear least square problem: Application of a general method , 1976 .

[44]  W. Häuβler,et al.  A Kantorovich-type convergence analysis for the Gauss-Newton-Method , 1986 .

[45]  Axel Ruhe.  Accelerated Gauss-Newton algorithms for nonlinear least squares problems , 1979 .

[46]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[47]  William C. Davidon,et al.  Optimally conditioned optimization algorithms without line searches , 1975, Math. Program..

[48]  L. Eldén.  Algorithms for the regularization of ill-conditioned least squares problems , 1977 .

[49]  R. Fletcher,et al.  Hybrid Methods for Nonlinear Least Squares , 1987 .

[50]  R. Fletcher.  A modified Marquardt subroutine for non-linear least squares , 1971 .

[51]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[52]  N. D. Villiers,et al.  A Continuation Method for Nonlinear Regression , 1981 .

[53]  J. L. Nazareth.  An Adaptive Method for Minimizing a Sum of Squares of Nonlinear Functions , 1983 .

[54]  L. Eldén.  A note on the computation of the generalized cross-validation function for ill-conditioned least squares problems , 1984 .

[55]  Jorge J. Moré,et al.  User Guide for Minpack-1 , 1980 .

[56]  M. R. Osborne.  Nonlinear least squares — the Levenberg algorithm revisited , 1976, The Journal of the Australian Mathematical Society. Series B. Applied Mathematics.

[57]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[58]  C. Fraley.  Computational Behavior of Gauss–Newton Methods , 1989 .

[59]  M. Powell.  CONVERGENCE PROPERTIES OF A CLASS OF MINIMIZATION ALGORITHMS , 1975 .

[60]  P. Wedin,et al.  A comparison of some algorithms for the nonlinear least squares problem , 1977 .

[61]  Shmuel S. Oren,et al.  On the selection of parameters in Self Scaling Variable Metric Algorithms , 1974, Math. Program..

[62]  R. Meyer.  Theoretical and Computational Aspects of Nonlinear Regression , 1970 .

[63]  J. Varah.  On Fitting Exponentials by Nonlinear Least Squares , 1982 .

[64]  John E. Dennis,et al.  A New Algorithm for Nonlinear Least Squares Curve Fitting , 1970 .

[65]  Philip E. Gill,et al.  Newton-type methods for unconstrained and linearly constrained optimization , 1974, Math. Program..

[66]  Per Lindström,et al.  A new linesearch algorithm for nonlinear least squares problems , 1984, Math. Program..

[67]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[68]  J. Stoer.  Principles of Sequential Quadratic Programming Methods for Solving Nonlinear Programs , 1985 .

[69]  M. Powell.  A New Algorithm for Unconstrained Optimization , 1970 .

[70]  C. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[71]  David M. Gay,et al.  Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust-Region Approach , 1983, TOMS.

[72]  P. Gill,et al.  Algorithms for the Solution of the Nonlinear Least-Squares Problem , 1978 .