Family of Projected Descent Methods for Optimization Problems with Simple Bounds

This paper presents a family of projected descent direction algorithms with inexact line search for solving large-scale minimization problems subject to simple bounds on the decision variables. The global convergence of algorithms in this family is ensured by conditions on the descent directions and line search. Whenever a sequence constructed by an algorithm in this family enters a sufficiently small neighborhood of a local minimizer ○ satisfying standard second-order sufficiency conditions, it gets trapped and converges to this local minimizer. Furthermore, in this case, the active constraint set at ○ is identified in a finite number of iterations. This fact is used to ensure that the rate of convergence to a local minimizer, satisfying standard second-order sufficiency conditions, depends only on the behavior of the algorithm in the unconstrained subspace. As a particular example, we present projected versions of the modified Polak–Ribière conjugate gradient method and the limited-memory BFGS quasi-Newton method that retain the convergence properties associated with those algorithms applied to unconstrained problems.

[1]  A. Goldstein.  Convex programming in Hilbert space , 1964 .

[2]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[3]  E. Polak,et al.  Constrained Minimization Problems in Finite-Dimensional Spaces , 1966 .

[4]  D. Luenberger.  Convergence rate of a penalty-function scheme , 1971 .

[5]  Elijah Polak,et al.  Computational methods in optimization , 1971 .

[6]  R. Sargent,et al.  On the convergence of sequential minimization algorithms , 1973 .

[7]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[8]  Edward J. Davison,et al.  Clipping-off gradient algorithms to compute optimal controls with constrained magnitude , 1974 .

[9]  D. Bertsekas.  Partial conjugate gradient methods for a class of optimal control problems , 1974 .

[10]  M. J. D. Powell,et al.  Restart procedures for the conjugate gradient method , 1977, Math. Program..

[11]  D. F. Shanno,et al.  Matrix conditioning and nonlinear optimization , 1978, Math. Program..

[12]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[13]  D. Bertsekas.  Projected Newton methods for optimization problems with simple constraints , 1981, 1981 20th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes.

[14]  D. Bertsekas.  Projected Newton methods for optimization problems with simple constraints , 1981, CDC 1981.

[15]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[16]  Gene H. Golub,et al.  Matrix computations , 1983 .

[17]  David G. Luenberger,et al.  Linear and nonlinear programming , 1984 .

[18]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[19]  J. Dunn.  A projected Newton method for minimization problems with nonlinear inequality constraints , 1988 .

[20]  P. Toint,et al.  Global convergence of a class of trust region algorithms for optimization with simple bounds , 1988 .

[21]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[22]  D. Bertsekas,et al.  Efficient dynamic programming implementations of Newton's method for unconstrained optimal control problems , 1989 .

[23]  J. J. Moré,et al.  Algorithms for bound constrained quadratic programming problems , 1989 .

[24]  P. Toint,et al.  A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds , 1991 .

[25]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[26]  François-Xavier Le Dimet,et al.  Numerical Experience with Limited-Memory Quasi-Newton and Truncated Newton Methods , 1993, SIAM J. Optim..

[27]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[28]  Ekkehard W. Sachs,et al.  Solution of Optimal Control Problems by a Pointwise Projected Newton Method , 1995 .

[29]  E. Polak,et al.  Consistent Approximations for Optimal Control Problems Based on Runge--Kutta Integration , 1996 .