Newton's method and its use in optimization

Abstract Newton’s method is a basic tool in numerical analysis and numerous applications, including operations research and data mining. We survey the history of the method, its main ideas, convergence results, modifications, its global behavior. We focus on applications of the method for various classes of optimization problems, such as unconstrained minimization, equality constrained problems, convex programming and interior point methods. Some extensions (non-smooth problems, continuous analog, Smale’s results, etc.) are discussed briefly, while some others (e.g., versions of the method to achieve global convergence) are addressed in more details.

[1]  R. Mifflin.  Semismooth and Semiconvex Functions in Constrained Optimization , 1977 .

[2]  Tjalling J. Ypma,et al.  Historical Development of the Newton-Raphson Method , 1995, SIAM Rev..

[3]  Boris Polyak.  Minimization of unsmooth functionals , 1969 .

[4]  Richard E. Ewing,et al.  "The Merging of Disciplines: New Directions in Pure, Applied, and Computational Mathematics" , 1986 .

[5]  Yurii Nesterov,et al.  Interior-point polynomial algorithms in convex programming , 1994, Siam studies in applied mathematics.

[6]  Aharon Ben-Tal,et al.  Lectures on modern convex optimization , 1987 .

[7]  Lawrence M. Graves,et al.  Some mapping theorems , 1950 .

[8]  V. Arnold.  SMALL DENOMINATORS AND PROBLEMS OF STABILITY OF MOTION IN CLASSICAL AND CELESTIAL MECHANICS , 1963 .

[9]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[10]  Michael F. Barnsley,et al.  Fractals everywhere , 1988 .

[11]  Ignacio E. Grossmann,et al.  Part II. Future perspective on optimization , 2004, Comput. Chem. Eng..

[12]  A. Goldstein.  On Newton's method , 1965 .

[13]  L. Kantorovich,et al.  Functional analysis in normed spaces , 1952 .

[14]  Heinz-Otto Peitgen,et al.  Newton's method and dynamical systems , 1989 .

[15]  Yurii Nesterov,et al.  Cubic regularization of a Newton scheme and its global performance , 2003 .

[16]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[17]  Werner C. Rheinboldt,et al.  Methods for solving systems of nonlinear equations , 1987 .

[18]  A. A. Bennett.  Newton's Method in General Analysis. , 1916, Proceedings of the National Academy of Sciences of the United States of America.

[19]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[20]  B. T. Polyak,et al.  CONVEXITY OF THE REACHABLE SET OF NONLINEAR SYSTEMS UNDER L 2 BOUNDED CONTROLS , 2022 .

[21]  J. J. Moré,et al.  Newton's Method , 1982 .

[22]  C. Sparrow.  The Fractal Geometry of Nature , 1984 .

[23]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[24]  L. Collatz.  Functional analysis and numerical mathematics , 1968 .

[25]  Boris Polyak.  Iterative methods using lagrange multipliers for solving extremal problems with constraints of the equation type , 1970 .

[26]  A. Ostrowski.  Solution of equations and systems of equations , 1967 .

[27]  L. Kantorovich,et al.  Functional analysis and applied mathematics , 1963 .

[28]  Steve Smale,et al.  Complexity theory and numerical analysis , 1997, Acta Numerica.

[29]  Boris Polyak.  Gradient methods for solving equations and inequalities , 1964 .

[30]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[31]  F. Franklin.  On Newton's Method of Approximation , .

[32]  Boris Polyak.  Convexity of Nonlinear Image of a Small Ball with Applications to Optimization , 2001 .

[33]  S. M. Robinson.  Newton's method for a class of nonsmooth functions , 1994 .

[34]  Boris Polyak.  The convexity principle and its applications , 2003 .

[35]  Liqun Qi,et al.  A nonsmooth version of Newton's method , 1993, Math. Program..

[36]  Stephen J. Wright.  Primal-Dual Interior-Point Methods , 1997, Other Titles in Applied Mathematics.

[37]  S. Smale.  Newton’s Method Estimates from Data at One Point , 1986 .

[38]  Ignacio E. Grossmann,et al.  Retrospective on optimization , 2004, Comput. Chem. Eng..

[39]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[40]  R. D. Murphy,et al.  Iterative solution of nonlinear equations , 1994 .

[41]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[42]  Alexander G. Ramm.  Acceleration of Convergence of A Continuous Analog of the Newton Method , 2002 .

[43]  Xiaojun Chen,et al.  Smoothing Methods and Semismooth Methods for Nondifferentiable Operator Equations , 2000, SIAM J. Numer. Anal..

[44]  Ambikeshwar Sharma.  On Newton's method of approximation , 1959 .

[45]  A. Ioffe,et al.  On the local surjection property , 1987 .

[46]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[47]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[48]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[49]  Dietmar Saupe,et al.  Cayley’s problem and Julia sets , 1984 .

[50]  Stephen M. Robinson,et al.  Perturbed Kuhn-Tucker points and rates of convergence for a class of nonlinear-programming algorithms , 1974, Math. Program..

[51]  S. Smale.  Convergent process of price adjust-ment and global newton methods , 1976 .

[52]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[53]  Diethard Klatte,et al.  Nonsmooth Equations in Optimization: "Regularity, Calculus, Methods And Applications" , 2006 .

[54]  Michael A. Saunders,et al.  SNOPT: An SQP Algorithm for Large-Scale Constrained Optimization , 2005, SIAM Rev..

[55]  S. Goldfeld,et al.  Maximization by Quadratic Hill-Climbing , 1966 .

[56]  M. A. Krasnoselʹskii,et al.  Approximate Solution of Operator Equations , 1972 .

[57]  Dennis Sullivan,et al.  On the iteration of a rational function: Computer experiments with Newton's method , 1983 .

[58]  Karol Baron.  On approximate solutions of a system of functional equations , 1983 .