Approximating Hessians in unconstrained optimization arising from discretized problems

We consider Hessian approximation schemes for large-scale unconstrained optimization in the context of discretized problems. The considered Hessians typically present a nontrivial sparsity and partial separability structure. This allows iterative quasi-Newton methods to solve them despite of their size. Structured finite-difference methods and updating schemes based on the secant equation are presented and compared numerically inside the multilevel trust-region algorithm proposed by Gratton et al. (IMA J. Numer. Anal. 28(4):827–861, 2008).

[1]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[2]  Serge Gratton,et al.  Numerical Experience with a Recursive Trust-Region Method for Multilevel Nonlinear Optimization , 2006 .

[3]  X. Yi.  On Automatic Differentiation , 2005 .

[4]  M. Powell,et al.  On the Estimation of Sparse Jacobian Matrices , 1974 .

[5]  T. Steihaug,et al.  Computing a sparse Jacobian matrix by rows and columns , 1998 .

[6]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[7]  Thomas F. Coleman,et al.  Software for estimating sparse Hessian matrices , 1985, TOMS.

[8]  S. Nash.  A survey of truncated-Newton methods , 2000 .

[9]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[10]  P. Toint,et al.  A recursive ℓ∞-trust-region method for bound-constrained nonlinear optimization , 2008 .

[11]  Serge Gratton,et al.  Numerical experience with a recursive trust-region method for multilevel nonlinear bound-constrained optimization , 2010, Optim. Methods Softw..

[12]  P. Toint,et al.  Optimal estimation of Jacobian and Hessian matrices that arise in finite difference calculations , 1984 .

[13]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[14]  Alex Pothen,et al.  What Color Is Your Jacobian? Graph Coloring for Computing Derivatives , 2005, SIAM Rev..

[15]  Philippe L. Toint,et al.  A note about sparsity exploiting quasi-Newton updates , 1981, Math. Program..

[16]  Andreas Griewank,et al.  On the unconstrained optimization of partially separable functions , 1982 .

[17]  William C. Davidon,et al.  Variable Metric Method for Minimization , 1959, SIAM J. Optim..

[18]  Shahadat Hossain,et al.  Sparsity issues in the computation of Jacobian matrices , 2002, ISSAC '02.

[19]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[20]  M. Powell,et al.  On the Estimation of Sparse Hessian Matrices , 1979 .

[21]  P. Toint.  On sparse and symmetric matrix updating subject to a linear equation , 1977 .

[22]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[23]  Griewank,et al.  On automatic differentiation , 1988 .

[24]  Charles A. Bouman,et al.  Multigrid algorithms for optimization and inverse problems , 2003, IS&T/SPIE Electronic Imaging.

[25]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[26]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[27]  DONALD GOLDFARB,et al.  A LINE SEARCH MULTIGRID METHOD FOR LARGE-SCALE CONVEX OPTIMIZATION , 2007 .

[28]  D. Sorensen.  An example concerning quasi-Newton estimation of a sparse hessian , 1981, SGNM.

[29]  Nicholas I. M. Gould,et al.  Lancelot: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A) , 1992 .

[30]  Thomas F. Coleman,et al.  Estimation of sparse hessian matrices and graph coloring problems , 1982, Math. Program..

[31]  S. Nash.  A multigrid approach to discretized optimization problems , 2000 .

[32]  K. Tanabe,et al.  Mathematical Programming: Recent Developments and Applications. , 1990 .

[33]  Serge Gratton,et al.  Recursive Trust-Region Methods for Multiscale Nonlinear Optimization , 2008, SIAM J. Optim..

[34]  Ravindra K. Ahuja,et al.  Network Flows: Theory, Algorithms, and Applications , 1993 .

[35]  P. Toint,et al.  A recursive trust-region method in infinity norm for bound-constrained nonlinear optimization , 2007 .

[36]  M. J. D. Powell,et al.  Nonlinear optimization, 1981 , 1982 .

[37]  P. Toint.  On the Superlinear Convergence of an Algorithm for Solving a Sparse Minimization Problem , 1979 .

[38]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .