A user's guide to nonlinear optimization algorithms

The purpose of this paper is to provide a user's introduction to the basic ideas currently favored in nonlinear optimization routines by numerical analysts. The primary focus will be on the unconstrained problem because the main ideas are much more settled. Although this is not a paper about nonlinear least squares, the rich structure of this important practical problem makes it a convenient example to illustrate the ideas we will discuss. We will make most use of this example in the first three sections which deal with the helpful concept of a local modeling technique and the attendant local convergence analysis. Stress will be put on ways used to improve a poor initial solution estimate, since this is one of the keys to choosing the most suitable routine for a particular application. This material is covered in the rather long Section IV. The discussion of the constrained problem in Section V will be a brief outline of the current issues involved in deciding what algorithms to implement. Section VI is devoted to some concluding remarks including sparse comments on large problems.

[1]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[2]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[3]  Philippe L. Toint,et al.  Forcing sparsity by projecting with respect to a non-diagonally weighted frobenius norm , 1983, Math. Program..

[4]  L. Grippo,et al.  A New Class of Augmented Lagrangians in Nonlinear Programming , 1979 .

[5]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[6]  J. Dennis,et al.  Two new unconstrained optimization algorithms which use function and gradient values , 1979 .

[7]  R. Tapia.  Diagonalized multiplier methods and quasi-Newton methods for constrained optimization , 1977 .

[8]  Richard H. Byrd,et al.  A Family of Trust Region Based Algorithms for Unconstrained Minimization with Strong Global Convergence Properties. , 1985 .

[9]  M. Powell,et al.  On the Estimation of Sparse Hessian Matrices , 1979 .

[10]  M. Powell.  A New Algorithm for Unconstrained Optimization , 1970 .

[11]  R. Tapia,et al.  QDN: a variable storage algorithm for unconstrained optimization , 1978 .

[12]  C. G. Broyden.  A Class of Methods for Solving Nonlinear Simultaneous Equations , 1965 .

[13]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[14]  Susana Gómez,et al.  The tunnelling method for solving the constrained global optimization problem with several non-connected feasible regions , 1982 .

[15]  John E. Dennis,et al.  Algorithm 573: NL2SOL—An Adaptive Nonlinear Least-Squares Algorithm [E4] , 1981, TOMS.

[16]  R. A. Tapia.  Augmented Lagrangian methods for constrained optimization: the role of the penalty constant , 1979 .

[17]  M. J. D. Powell,et al.  Algorithms for nonlinear constraints that use lagrangian functions , 1978, Math. Program..

[18]  S. Goldfeld,et al.  Maximization by Quadratic Hill-Climbing , 1966 .

[19]  Olvi L. Mangasarian,et al.  Nonlinear Programming , 1969 .

[20]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[21]  N. Maratos,et al.  Exact penalty function algorithms for finite dimensional and control optimization problems , 1978 .

[22]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[23]  Shih-Ping Han.  A globally convergent method for nonlinear programming , 1975 .

[24]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[25]  E. Allgower,et al.  Simplicial and Continuation Methods for Approximating Fixed Points and Solutions to Systems of Equations , 1980 .

[26]  P. Boggs,et al.  Augmented Lagrangians which are quadratic in the multiplier , 1980 .

[27]  Dianne P. O'Leary,et al.  A discrete Newton algorithm for minimizing a function of many variables , 1982, Math. Program..

[28]  Ph. L. Toint,et al.  A sparse quasi-Newton update derived variationally with a nondiagonally weighted Frobenius norm , 1981 .

[29]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[30]  Danny C. Sorensen,et al.  On the use of directions of negative curvature in a modified newton method , 1979, Math. Program..

[31]  A. V. Levy,et al.  Topics in global optimization , 1982 .

[32]  David M. author-Gay.  Computing Optimal Locally Constrained Steps , 1981 .

[33]  H. Walker,et al.  Inaccuracy in quasi-Newton methods: Local improvement theorems , 1984 .

[34]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[35]  A. Goldstein,et al.  Optimization of functions whose values are subject to small errors , 1977 .

[36]  Leon S. Lasdon,et al.  The Status of Nonlinear Programming Software , 1979, Oper. Res..

[37]  Homer F. Walker,et al.  Least-Change Sparse Secant Update Methods with Inaccurate Secant Conditions , 1985 .

[38]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[39]  L. Watson.  Engineering applications of the Chow-Yorke algorithm , 1981 .

[40]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[41]  J. J. Moré,et al.  Newton's Method , 1982 .

[42]  David F. Shanno,et al.  Conjugate Gradient Methods with Inexact Searches , 1978, Math. Oper. Res..

[43]  M. J. D. Powell,et al.  On the global convergence of trust region algorithms for unconstrained minimization , 1984, Math. Program..

[44]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[45]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[46]  L. Watson.  An Algorithm That is Globally Convergent with Probability One for a Class of Nonlinear Two-Point Boundary Value Problems , 1979 .

[47]  M. Powell.  CONVERGENCE PROPERTIES OF A CLASS OF MINIMIZATION ALGORITHMS , 1975 .

[48]  T. Ypma.  The Effect of Rounding Errors on Newton-like Methods , 1983 .

[49]  M GayDavid,et al.  Algorithm 611: Subroutines for Unconstrained Minimization Using a Model/Trust-Region Approach , 1983 .

[50]  R. Fletcher.  A modified Marquardt subroutine for non-linear least squares , 1971 .

[51]  D. Sorensen.  Newton's method with a model trust region modification , 1982 .

[52]  Thomas F. Coleman,et al.  Estimation of sparse hessian matrices and graph coloring problems , 1982, Math. Program..

[53]  S. Glad.  Properties of updating methods for the multipliers in augmented Lagrangians , 1979 .

[54]  Francesco Zirilli,et al.  The Solution of Nonlinear Systems of Equations by Second Order Systems of O.D.E. and Linearly Implicit A-Stable Techniques , 1982 .

[55]  Jorge J. Moré,et al.  Algorithm 566: FORTRAN Subroutines for Testing Unconstrained Optimization Software [C5], [E4] , 1981, TOMS.

[56]  P. Boggs,et al.  A stability analysis for perturbed nonlinear iterative methods , 1976 .

[57]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .