Trust-Region Interior-Point SQP Algorithms for a Class of Nonlinear Programming Problems

In this paper, a family of trust-region interior-point sequential quadratic programming (SQP) algorithms for the solution of a class of minimization problems with nonlinear equality constraints and simple bounds on some of the variables is described and analyzed. Such nonlinear programs arise, e.g., from the discretization of optimal control problems. The algorithms treat states and controls as independent variables. They are designed to take advantage of the structure of the problem. In particular they do not rely on matrix factorizations of the linearized constraints but use solutions of the linearized state equation and the adjoint equation. They are well suited for large scale problems arising from optimal control problems governed by partial differential equations. The algorithms keep strict feasibility with respect to the bound constraints by using an affine scaling method proposed, for a different class of problems, by Coleman and Li [SIAM J. Optim., 6 (1996), pp. 418--445] and they exploit trust-region techniques for equality-constrained optimization. Thus, they allow the computation of the steps using a variety of methods, including many iterative techniques. Global convergence of these algorithms to a first-order Karush--Kuhn--Tucker (KKT) limit point is proved under very mild conditions on the trial steps. Under reasonable, but more stringent, conditions on the quadratic model and on the trial steps, the sequence of iterates generated by the algorithms is shown to have a limit point satisfying the second-order necessary KKT conditions. The local rate of convergence to a nondegenerate strict local minimizer is q-quadratic. The results given here include, as special cases, current results for only equality constraints and for only simple bounds. Numerical results for the solution of an optimal control problem governed by a nonlinear heat equation are reported.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[3]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[4]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[5]  I. M. Gel'fand,et al.  Some problems in the theory of quasilinear equations , 1987 .

[6]  L. Armijo.  Minimization of functions having Lipschitz continuous first partial derivatives. , 1966 .

[7]  S. Goldfeld,et al.  Maximization by Quadratic Hill-Climbing , 1966 .

[8]  M. Powell.  A New Algorithm for Unconstrained Optimization , 1970 .

[9]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[10]  P. Wolfe.  Convergence Conditions for Ascent Methods. II: Some Corrections , 1971 .

[11]  C. Reinsch.  Smoothing by spline functions. II , 1971 .

[12]  Elijah Polak,et al.  Computational methods in optimization , 1971 .

[13]  Thomas Simpson.  Essays on several curious and useful subjects, in speculative and mix'd mathematics : illustrated by a variety of examples , 1972 .

[14]  E. Polak,et al.  Computational methods in optimization : a unified approach , 1972 .

[15]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[16]  S. W. Thomas.  Sequential estimation techniques for quasi-newton algorithms. , 1975 .

[17]  M. Powell.  CONVERGENCE PROPERTIES OF A CLASS OF MINIMIZATION ALGORITHMS , 1975 .

[18]  C. W. Groetsch,et al.  Generalized inverses of linear operators , 1977 .

[19]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[20]  J. Dennis,et al.  Two new unconstrained optimization algorithms which use function and gradient values , 1979 .

[21]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .

[22]  Ekkehard W. Sachs,et al.  Algorithmic methods in optimal control , 1981 .

[23]  David M. author-Gay.  Computing Optimal Locally Constrained Steps , 1981 .

[24]  Philip E. Gill,et al.  Practical optimization , 1981 .

[25]  Richard H. Byrd,et al.  A Family of Trust Region Based Algorithms for Unconstrained Minimization with Strong Global Convergence Properties. , 1985 .

[26]  D. Sorensen.  Newton's method with a model trust region modification , 1982 .

[27]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[28]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[29]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[30]  Trond Steihaug,et al.  Truncated-newtono algorithms for large-scale unconstrained optimization , 1983, Math. Program..

[31]  Trond Steihaug.  Local and superlinear convergence for truncated iterated projections methods , 1983, Math. Program..

[32]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[33]  Michael A. Saunders,et al.  User''s guide for NPSOL (Ver-sion 4.0): A FORTRAN package for nonlinear programming , 1984 .

[34]  M. J. D. Powell,et al.  On the global convergence of trust region algorithms for unconstrained minimization , 1984, Math. Program..

[35]  Danny C. Sorensen,et al.  A note on the computation of an orthonormal basis for the null space of a matrix , 1982, Math. Program..

[36]  Richard A. Tapia,et al.  A trust region strategy for nonlinear equality constrained op-timization , 1984 .

[37]  S. Nash.  Newton-Type Minimization via the Lanczos Method , 1984 .

[38]  A. Vardi.  A Trust Region Algorithm for Equality Constrained Minimization: Convergence Properties and Implementation , 1985 .

[39]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[40]  Michael A. Saunders,et al.  Properties of a representation of a basis for the null space , 1985, Math. Program..

[41]  Leon S. Lasdon,et al.  An Improved Successive Linear Programming Algorithm , 1985 .

[42]  H. Keller,et al.  Continuation-Conjugate Gradient Methods for the Least Squares Solution of Nonlinear Boundary Value Problems , 1985 .

[43]  Jonathan Goodman,et al.  Newton's method for constrained optimization , 1985, Math. Program..

[44]  Richard H. Byrd,et al.  A Trust Region Algorithm for Nonlinearly Constrained Optimization , 1987 .

[45]  Richard H. Byrd,et al.  Continuity of the null space basis and constrained optimization , 1986, Math. Program..

[46]  P. Gill,et al.  Some theoretical properties of an augmented lagrangian merit function , 1986 .

[47]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[48]  Jong-Shi Pang,et al.  Inexact Newton methods for the nonlinear complementarity problem , 1986, Math. Program..

[49]  Klaus Schittkowski,et al.  NLPQL: A fortran subroutine solving constrained nonlinear programming problems , 1986 .

[50]  Harry Yserentant,et al.  On the multi-level splitting of finite element spaces , 1986 .

[51]  C. Kelley.  Iterative Methods for Linear and Nonlinear Equations , 1987 .

[52]  P. Brown.  A local convergence theory for combined inexact-Newton/finite-difference projection methods , 1987 .

[53]  P. Toint.  Global Convergence of a a of Trust-Region Methods for Nonconvex Minimization in Hilbert Space , 1988 .

[54]  Richard H. Byrd,et al.  Approximate solution of the trust region problem by minimization over two-dimensional subspaces , 1988, Math. Program..

[55]  J. Dennis,et al.  A global convergence theory for a class of trust region algorithms for constrained optimization , 1988 .

[56]  R. Schnabel,et al.  A view of unconstrained optimization , 1989 .

[57]  E. Omojokun.  Trust region algorithms for optimization with nonlinear equality and inequality constraints , 1990 .

[58]  Yousef Saad,et al.  Hybrid Krylov Methods for Nonlinear Systems of Equations , 1990, SIAM J. Sci. Comput..

[59]  Gerardo Toraldo,et al.  Convergence properties of trust region methods for linear and convex constraints , 1990, Math. Program..

[60]  K. Kunisch,et al.  The augmented lagrangian method for parameter estimation in elliptic systems , 1990 .

[61]  J. Z. Zhang,et al.  Projected quasi-Newton algorithm with trust region for constrained optimization , 1990 .

[62]  Ya-Xiang Yuan,et al.  On a subproblem of trust region algorithms for constrained optimization , 1990, Math. Program..

[63]  G. Golub,et al.  Quadratically constrained least squares and quadratic problems , 1991 .

[64]  M. El-Alem.  A global convergence theory for the Celis-Dennis-Tapia trust-region algorithm for constrained optimization , 1991 .

[65]  P. Toint,et al.  A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds , 1991 .

[66]  Ya-Xiang Yuan,et al.  A trust region algorithm for equality constrained optimization , 1990, Math. Program..

[67]  E. Sachs,et al.  A prospective look at SQP methods for semilinear parabolic control problems , 1991 .

[68]  Jorge Nocedal,et al.  A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization , 1991, SIAM J. Optim..

[69]  J. Burger,et al.  Functional and numerical solution of a control problem originating from heat transfer , 1991 .

[70]  Karl Kunisch,et al.  Estimation of a temporally and spatially varying diffusion coefficient in a parabolic system by an augmented Lagrangian technique , 1991 .

[71]  Peter Deuflhard,et al.  Global inexact Newton methods for very large scale nonlinear problems , 1991, IMPACT Comput. Sci. Eng..

[72]  R. Carter.  On the global convergence of trust region algorithms using inexact gradient information , 1991 .

[73]  Stephen J. Wright,et al.  Sequential quadratic programming for certain parameter identification problems , 1991, Math. Program..

[74]  W. Symes,et al.  Conjugate residual methods for almost symmetric linear systems , 1992 .

[75]  Matthias Heinkenschloss.  On the solution of a two ball trust region subproblem , 1992, Universität Trier, Mathematik/Informatik, Forschungsbericht.

[76]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[77]  Yin Zhang,et al.  Computing a Celis-Dennis-Tapia trust-region step for equality constrained optimization , 1992, Math. Program..

[78]  C. M. Samuelsen.  The Dikin-Karmarkar Principle for Steepest Descent , 1992 .

[79]  James V. Burke,et al.  A Robust Trust Region Method for Constrained Nonlinear Programming Problems , 1992, SIAM J. Optim..

[80]  Karl Kunisch,et al.  Reduced SQP methods for parameter identification problems , 1992 .

[81]  Ekkehard W. Sachs,et al.  Numerical solution of a nonlinear parabolic control problem by a reduced SQP method , 1992, Comput. Optim. Appl..

[82]  Yuanfu Xie.  Reduced Hessian algorithms for solving large-scale equality constrained optimization problems , 1992 .

[83]  M. El Hallabi.  A Global Convergence Theory for Arbitrary Norm Trust-Region Algorithms for Equality Constrained Optimization , 1993 .

[84]  Nicholas I. M. Gould,et al.  Global Convergence of a Class of Trust Region Algorithms for Optimization Using Inexact Projections on Convex Constraints , 1993, SIAM J. Optim..

[85]  Stephen J. Wright.  Interior point methods for optimal control of discrete time systems , 1993 .

[86]  N. Alexandrov.  Multilevel algorithms for nonlinear equations and equality constrained optimization , 1993 .

[87]  Roland W. Freund,et al.  A Transpose-Free Quasi-Minimal Residual Algorithm for Non-Hermitian Linear Systems , 1993, SIAM J. Sci. Comput..

[88]  J. J. Moré.  Generalizations of the trust region problem , 1993 .

[89]  T. Plantenga.  Large-scale nonlinear constrained optimization using trust regions , 1994 .

[90]  J. Betts,et al.  A sparse nonlinear optimization algorithm , 1994 .

[91]  A simultaneous approach for flowsheet optimization with existing modelling procedures : Process design , 1994 .

[92]  Yuying Li.  A Trust Region and Affine Scaling Method for Nonlinearly Constrained Minimization , 1994 .

[93]  Yousef Saad,et al.  Convergence Theory of Nonlinear Newton-Krylov Algorithms , 1994, SIAM J. Optim..

[94]  John E. Dennis,et al.  Problem Formulation for Multidisciplinary Optimization , 1994, SIAM J. Optim..

[95]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[96]  Jorge Nocedal,et al.  Representations of quasi-Newton matrices and their use in limited memory methods , 1994, Math. Program..

[97]  Thomas F. Coleman,et al.  On the convergence of interior-reflective Newton methods for nonlinear minimization subject to bounds , 1994, Math. Program..

[98]  Ekkehard W. Sachs,et al.  Numerical Solution of Parabolic State Constrained Control Problems Using SQP- and Interior-Point-Methods , 1994 .

[99]  Richard Barrett,et al.  Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods , 1994, Other Titles in Applied Mathematics.

[100]  Homer F. Walker,et al.  Globally Convergent Inexact Newton Methods , 1994, SIAM J. Optim..

[101]  Francisco J. Prieto,et al.  A Sequential Quadratic Programming Algorithm Using an Incomplete Solution of the Subproblem , 1995, SIAM J. Optim..

[102]  G. Reinelt,et al.  Fast Recursive SQP MethodsforLarge-Scale Optimal Control Problems , 1995 .

[103]  Jorge Nocedal,et al.  A Reduced Hessian Method for Large-Scale Constrained Optimization , 1995, SIAM J. Optim..

[104]  What Happens When We Trust a Region That is a Line , 1995 .

[105]  Paul T. Boggs,et al.  Sequential Quadratic Programming , 1995, Acta Numerica.

[106]  Mahmoud El-Alem,et al.  A Robust Trust-Region Algorithm with a Nonmonotonic Penalty Parameter Scheme for Constrained Optimization , 1995, SIAM J. Optim..

[107]  M. Heinkenschloss,et al.  ANALYSIS OF INEXACT TRUST – REGION INTERIOR – POINT SQP ALGORITHMS , 1995 .

[108]  Ekkehard W. Sachs,et al.  Solution of Optimal Control Problems by a Pointwise Projected Newton Method , 1995 .

[109]  Henry Wolkowicz,et al.  Indefinite Trust Region Subproblems and Nonsymmetric Eigenvalue Perturbations , 1995, SIAM J. Optim..

[110]  José Mario Martínez,et al.  A trust-region strategy for minimization on arbitrary domains , 1995, Math. Program..

[111]  Ya-Xiang Yuan,et al.  On the convergence of a new trust region algorithm , 1995 .

[112]  Tjalling J. Ypma,et al.  Historical Development of the Newton-Raphson Method , 1995, SIAM Rev..

[113]  Matthias Heinkenschloss.  Projected Sequential Quadratic Programming Methods , 1996, SIAM J. Optim..

[114]  L. N. Vicente,et al.  Trust-Region Interior-Point Algorithms for Minimization Problems with Simple Bounds , 1996 .

[115]  T. Tsuchiya,et al.  On the formulation and theory of the Newton interior-point method for nonlinear programming , 1996 .

[116]  Homer F. Walker,et al.  Choosing the Forcing Terms in an Inexact Newton Method , 1996, SIAM J. Sci. Comput..

[117]  Thomas F. Coleman,et al.  An Interior Trust Region Approach for Nonlinear Minimization Subject to Bounds , 1993, SIAM J. Optim..

[118]  M. El-Alem.  Convergence to a second-order point of a trust-region algorithm with a nonmonotonic penalty parameter for constrained optimization , 1996 .

[119]  F.-S. Kupfer,et al.  An Infinite-Dimensional Convergence Theory for Reduced SQP Methods in Hilbert Space , 1996, SIAM J. Optim..

[120]  Danny C. Sorensen,et al.  Minimization of a Large-Scale Quadratic FunctionSubject to a Spherical Constraint , 1997, SIAM J. Optim..

[121]  Walter Murray,et al.  Sequential Quadratic Programming Methods for Large-Scale Problems , 1997, Comput. Optim. Appl..

[122]  L. Biegler,et al.  A MULTIPLIER-FREE, REDUCED HESSIAN METHOD FOR PROCESS OPTIMIZATION , 1997 .

[123]  John E. Dennis,et al.  On the Convergence Theory of Trust-Region-Based Algorithms for Equality-Constrained Optimization , 1997, SIAM J. Optim..

[124]  J. Frédéric Bonnans,et al.  A Trust Region Interior Point Algorithm for Linearly Constrained Optimization , 1997, SIAM J. Optim..

[125]  Franz Rendl,et al.  A semidefinite framework for trust region subproblems with applications to large scale minimization , 1997, Math. Program..

[126]  M. Heinkenschloss,et al.  An Optimal Control Problem for Flows with Discontinuities , 1997 .

[127]  John E. Dennis,et al.  A Global Convergence Theory for General Trust-Region-Based Algorithms for Equality Constrained Optimization , 1997, SIAM J. Optim..

[128]  Hiroshi Yamashita.  A globally convergent primal-dual interior point method for constrained optimization , 1998 .

[129]  Jorge Nocedal,et al.  On the Implementation of an Algorithm for Large-Scale Equality Constrained Optimization , 1998, SIAM J. Optim..

[130]  Luis S. Vicente.  On interior-point Newton algorithms for discretized optimal control problems with state constraints ∗ , 1998 .

[131]  J. B. Rosen,et al.  SQP Methods and their Application to Numerical Optimal Control , 1998 .

[132]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .