Representations of quasi-Newton matrices and their use in limited memory methods

We derive compact representations of BFGS and symmetric rank-one matrices for optimization. These representations allow us to efficiently implement limited memory methods for large constrained optimization problems. In particular, we discuss how to compute projections of limited memory matrices onto subspaces. We also present a compact representation of the matrices generated by Broyden's update for solving systems of nonlinear equations.

[1]  C. G. Broyden.  A Class of Methods for Solving Nonlinear Simultaneous Equations , 1965 .

[2]  John Grover Barnes,et al.  An Algorithm for Solving Non-Linear Equations Based on the Secant Method , 1965, Comput. J..

[3]  G. R. Walsh,et al.  Methods Of Optimization , 1976 .

[4]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[5]  P. Gill,et al.  Quasi-Newton Methods for Unconstrained Optimization , 1972 .

[6]  R. Schnabel,et al.  Solving Systems of Non-Linear Equations by Broyden&Apos;S Method with Projected Updates , 1977 .

[7]  G. Strang,et al.  The solution of nonlinear finite element equations , 1979 .

[8]  J. Nocedal.  Updating Quasi-Newton Matrices With Limited Storage , 1980 .

[9]  Albert G. Buckley,et al.  QN-like variable storage conjugate gradients , 1983, Math. Program..

[10]  R. Schnabel.  Quasi-Newton Methods Using Multiple Secant Equations. , 1983 .

[11]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[12]  P. Toint,et al.  Testing a class of methods for solving minimization problems with simple bounds on the variables , 1988 .

[13]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[14]  H. Walker.  Implementation of the GMRES method using householder transformations , 1988 .

[15]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[16]  Claude Lemaréchal,et al.  Some numerical experiments with variable-storage quasi-Newton algorithms , 1989, Math. Program..

[17]  Jean Charles Gilbert,et al.  Automatic differentiation and the step computation in the limited memory BFGS method , 1993 .

[18]  Richard H. Byrd,et al.  A Theoretical and Experimental Study of the Symmetric Rank-One Update , 1993, SIAM J. Optim..

[19]  R. D. Murphy,et al.  Iterative solution of nonlinear equations , 1994 .

[20]  Jorge Nocedal,et al.  A Reduced Hessian Method for Large-Scale Constrained Optimization , 1995, SIAM J. Optim..

[21]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..