A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization

This paper examines the numerical performances of two methods for large-scale optimization: a limited memory quasi-Newton method (L-BFGS), and a discrete truncated-Newton method (TN). Various ways of classifying test problems are discussed in order to better understand the types of problems that each algorithm solves well. The L-BFGS and TN methods are also compared with the Polak–Ribiere conjugate gradient method.

[1]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[2]  Shmuel S. Oren,et al.  Optimal conditioning of self-scaling variable Metric algorithms , 1976, Math. Program..

[3]  Charles B. Dunham,et al.  Remark on “Algorithm 500: Minimization of Unconstrained Multivariate Functions [E4]” , 1977, TOMS.

[4]  P. Gill,et al.  Conjugate-Gradient Methods for Large-Scale Nonlinear Optimization. , 1979 .

[5]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[6]  P. Gill,et al.  Methods for Large-Scale Nonlinear Optimization. , 1980 .

[7]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[8]  Dianne P. O'Leary,et al.  A discrete Newton algorithm for minimizing a function of many variables , 1982, Math. Program..

[9]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .

[10]  Albert G. Buckley,et al.  QN-like variable storage conjugate gradients , 1983, Math. Program..

[11]  Thomas F. Coleman,et al.  Estimation of sparse hessian matrices and graph coloring problems , 1982, Math. Program..

[12]  S. Nash.  Preconditioning of Truncated-Newton Methods , 1985 .

[13]  Griewank,et al.  On automatic differentiation , 1988 .

[14]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[15]  Claude Lemaréchal,et al.  Some numerical experiments with variable-storage quasi-Newton algorithms , 1989, Math. Program..

[16]  Stephen G. Nash,et al.  Block truncated-Newton methods for parallel optimization , 1989, Math. Program..

[17]  S. Nash,et al.  Assessing a search direction within a truncated-newton method , 1990 .