Projected gradient methods for linearly constrained problems

The aim of this paper is to study the convergence properties of the gradient projection method and to apply these results to algorithms for linearly constrained problems. The main convergence result is obtained by defining a projected gradient, and proving that the gradient projection method forces the sequence of projected gradients to zero. A consequence of this result is that if the gradient projection method converges to a nondegenerate point of a linearly constrained problem, then the active and binding constraints are identified in a finite number of iterations. As an application of our theory, we develop quadratic programming algorithms that iteratively explore a subspace defined by the active constraints. These algorithms are able to drop and add many constraints from the active set, and can either compute an accurate minimizer by a direct method, or an approximate minimizer by an iterative method of the conjugate gradient type. Thus, these algorithms are attractive for large scale problems. We show that it is possible to develop a finite terminating quadratic programming algorithm without non-degeneracy assumptions.

[1]  A. Goldstein.  Convex programming in Hilbert space , 1964 .

[2]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[3]  Boris Polyak.  The conjugate gradient method in extremal problems , 1969 .

[4]  E. H. Zarantonello.  Projections on Convex Sets in Hilbert Space and Spectral Theory: Part I. Projections on Convex Sets: Part II. Spectral Theory , 1971 .

[5]  G. McCormick,et al.  The Gradient Projection Method under Mild Differentiability Conditions , 1972 .

[6]  E. A. Nurminskii.  Convergence of the gradient projection method , 1973 .

[7]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[8]  D. O’Leary.  A generalized conjugate gradient algorithm for solving a class of quadratic programming problems , 1977 .

[9]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[10]  J. Dunn.  Global and Asymptotic Convergence Rate Estimates for a Class of Projected Gradient Processes , 1981 .

[11]  D. Bertsekas.  Projected Newton methods for optimization problems with simple constraints , 1981, 1981 20th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes.

[12]  Philip E. Gill,et al.  Practical optimization , 1981 .

[13]  T. M. Williams.  Practical Methods of Optimization. Vol. 2 — Constrained Optimization , 1982 .

[14]  D. Bertsekas,et al.  CONVERGENCE OF A GRADIENT PROJECTION METHOD , 1982 .

[15]  D. Bertsekas,et al.  TWO-METRIC PROJECTION METHODS FOR CONSTRAINED OPTIMIZATION* , 1984 .

[16]  R. Phelps.  The gradient projection method using Curry's steplength , 1986 .

[17]  J. Dunn.  On the convergence of projected gradient processes to singular critical points , 1987 .

[18]  J. J. Moré,et al.  On the identification of active constraints , 1988 .