Enriched Methods for Large-Scale Unconstrained Optimization

This paper describes a class of optimization methods that interlace iterations of the limited memory BFGS method (L-BFGS) and a Hessian-free Newton method (HFN) in such a way that the information collected by one type of iteration improves the performance of the other. Curvature information about the objective function is stored in the form of a limited memory matrix, and plays the dual role of preconditioning the inner conjugate gradient iteration in the HFN method and of providing an initial matrix for L-BFGS iterations. The lengths of the L-BFGS and HFN cycles are adjusted dynamically during the course of the optimization. Numerical experiments indicate that the new algorithms are both effective and not sensitive to the choice of parameters.

[1]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[2]  Dianne P. O'Leary,et al.  A discrete Newton algorithm for minimizing a function of many variables , 1982, Math. Program..

[3]  Trond Steihaug,et al.  Truncated-newtono algorithms for large-scale unconstrained optimization , 1983, Math. Program..

[4]  S. Nash.  Preconditioning of Truncated-Newton Methods , 1985 .

[5]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[6]  Claude Lemaréchal,et al.  Some numerical experiments with variable-storage quasi-Newton algorithms , 1989, Math. Program..

[7]  Jorge Nocedal,et al.  A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization , 1991, SIAM J. Optim..

[8]  P. Toint,et al.  Lancelot: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A) , 1992 .

[9]  Tamar Schlick,et al.  TNPACK—a truncated Newton minimization package for large-scale problems: II. Implementation examples , 1992, TOMS.

[10]  Nicholas I. M. Gould,et al.  CUTE: constrained and unconstrained testing environment , 1995, TOMS.

[11]  J. Nocedal,et al.  TOWARDS A DISCRETE NEWTON METHOD WITH MEMORY FOR LARGE(cid:1)SCALE OPTIMIZATION (cid:1) , 1996 .

[12]  Luis Morales,et al.  Algorithm PREQN Fortran Subroutines for Preconditioning the Conjugate Gradient Method , 1999 .

[13]  Tamar Schlick,et al.  Efficient Implementation of the Truncated-Newton Algorithm for Large-Scale Chemistry Applications , 1999, SIAM J. Optim..

[14]  Chih-Jen Lin,et al.  Newton's Method for Large Bound-Constrained Optimization Problems , 1999, SIAM J. Optim..

[15]  Tamar Schlick,et al.  Remark on Algorithm 702—the updated truncated Newton minimization package , 1999, TOMS.

[16]  Jorge Nocedal,et al.  Automatic Preconditioning by Limited Memory Quasi-Newton Updating , 1999, SIAM J. Optim..

[17]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[18]  Jorge Nocedal,et al.  Algorithm 809: PREQN: Fortran 77 subroutines for preconditioning the conjugate gradient method , 2001, TOMS.

[19]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..