Newton's method with a model trust region modification

A modified Newton method for unconstrained minimization is presented and analyzed. The modification is based upon the model trust region approach. This report contains a thorough analysis of the locally constrained quadratic minimizations that arise as subproblems in the modified Newton iteration. Several promising alternatives are presented for solving these subproblems in ways that overcome certain theoretical difficulties exposed by this analysis. Very strong convergence results are presented concerning the minimization algorithm. In particular, the explicit use of second order information is justified by demonstrating that the iterates converge to a point which satisfies the second order necessary conditions for minimization. With the exception of very pathological cases this occurs whenever the algorithm is applied to problems with continuous second partial derivatives.

[1]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[2]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[3]  G. Forsythe,et al.  On the Stationary Values of a Second-Degree Polynomial on the Unit Sphere , 1965 .

[4]  J. H. Wilkinson.  The algebraic eigenvalue problem , 1966 .

[5]  S. Goldfeld,et al.  Maximization by Quadratic Hill-Climbing , 1966 .

[6]  C. Reinsch.  Smoothing by spline functions , 1967 .

[7]  M. J. D. Powell,et al.  Nonlinear Programming—Sequential Unconstrained Minimization Techniques , 1969 .

[8]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[9]  J. Bunch,et al.  Direct Methods for Solving Symmetric Indefinite Systems of Linear Equations , 1971 .

[10]  J. O. Aasen.  On the reduction of a symmetric matrix to tridiagonal form , 1971 .

[11]  R. Fletcher.  A modified Marquardt subroutine for non-linear least squares , 1971 .

[12]  G. Stewart.  Introduction to matrix computations , 1973 .

[13]  M. D. Hebden,et al.  An algorithm for minimization using exact second derivatives , 1973 .

[14]  Philip E. Gill,et al.  Newton-type methods for unconstrained and linearly constrained optimization , 1974, Math. Program..

[15]  S. W. Thomas.  Sequential estimation techniques for quasi-newton algorithms. , 1975 .

[16]  M. Powell.  CONVERGENCE PROPERTIES OF A CLASS OF MINIMIZATION ALGORITHMS , 1975 .

[17]  Jean-Philippe Vial,et al.  Unconstrained Optimization by Approximation of the Gradient Path , 1977, Math. Oper. Res..

[18]  R. Fletcher,et al.  A modified Newton method for minimization , 1977 .

[19]  Garth P. McCormick,et al.  A modification of Armijo's step-size rule for negative curvature , 1977, Math. Program..

[20]  J. J. Moré,et al.  Levenberg--Marquardt algorithm: implementation and theory , 1977 .

[21]  W. Gander.  On the linear least squares problem with a quadratic constraint , 1978 .

[22]  E. Polak,et al.  A second-order method for unconstrained optimization , 1978 .

[23]  S. Kaniel,et al.  A Modified Newton's Method for Unconstrained Minimization , 1979 .

[24]  Danny C. Sorensen,et al.  On the use of directions of negative curvature in a modified newton method , 1979, Math. Program..

[25]  Donald Goldfarb,et al.  Curvilinear path steplength algorithms for minimization which use directions of negative curvature , 1980, Math. Program..