Preconditioned conjugate gradient algorithm for large scale problems with box constraints

The paper describes a new conjugate gradient algorithm for large scale nonconvex problems with box constraints. In order to speed up the convergence the algorithm employs a scaling matrix which transforms the space of original variables into the space in which Hessian matrices of functionals describing the problems have more clustered eigenvalues. This is done efficiently by applying limited memory BFGS updating matrices. Once the scaling matrix is calculated, the next few iterations of the conjugate gradient algorithms are performed in the transformed space. The box constraints are treated efficiently by the projection. We believe that the preconditioned conjugate gradient algorithm is competitive to the LBFGS-B algorithm. We give some numerical results which support our claim.

[1]  R. Pytlak,et al.  An Efficient Algorithm for Large-Scale Nonlinear Programming Problems with Simple Bounds on the Variables , 1998, SIAM J. Optim..

[2]  Jorge Nocedal,et al.  Representations of quasi-Newton matrices and their use in limited memory methods , 1994, Math. Program..

[3]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[4]  P. Toint,et al.  Testing a class of methods for solving minimization problems with simple bounds on the variables , 1988 .

[5]  Tomasz Tarnawski,et al.  Preconditioned conjugate gradient algorithms for nonconvex problems , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[6]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[7]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[8]  R. Pytlak.  On the convergence of conjugate gradient algorithms , 1994 .

[9]  James V. Burke,et al.  Exposing Constraints , 1994, SIAM J. Optim..

[10]  D. Bertsekas,et al.  Projected Newton methods and optimization of multicommodity flows , 1982, 1982 21st IEEE Conference on Decision and Control.

[11]  P. Wolfe.  Note on a method of conjugate subgradients for minimizing nondifferentiable functions , 1974 .

[12]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[13]  C. Lemaréchal.  An extension of davidon methods to non differentiable problems , 1975 .

[14]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[15]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[16]  Gene H. Golub,et al.  Matrix computations , 1983 .

[17]  Jorge Nocedal,et al.  Global Convergence Properties of Conjugate Gradient Methods for Optimization , 1992, SIAM J. Optim..

[18]  D. Bertsekas.  Projected Newton methods for optimization problems with simple constraints , 1981, CDC 1981.

[19]  J. J. Moré,et al.  On the identification of active constraints , 1988 .

[20]  P. Toint,et al.  Global convergence of a class of trust region algorithms for optimization with simple bounds , 1988 .

[21]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.