Sparse Matrix Methods in Optimization

Optimization algorithms typically require the solution of many systems of linear equations $B_k y_k = b_k $. When large numbers of variables or constraints are present, these linear systems could account for much of the total computation time.Both direct and iterative equation solvers are needed in practice. Unfortunately, most of the off-the-shelf solvers are designed for single systems, whereas optimization problems give rise to hundreds or thousands of systems. To avoid refactorization, or to speed the convergence of an iterative method, it is essential to note that $B_k $ is related to $B_{k - 1} $.We review various sparse matrices that arise in optimization, and discuss compromises that are currently being made in dealing with them. Since significant advances continue to be made with single-system solvers, we give special attention to methods that allow such solvers to be used repeatedly on a sequence of modified systems (e.g., the product-form update; use of the Schur complement). The speed of factorizing a matrix then becomes relatively less important than the efficiency of subsequent solves with very many right-hand sides.At the same time, we hope that future improvements to linear-equation software will be oriented more specifically to the case of related matrices $B_k $.

[1]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[2]  G. Dantzig,et al.  THE PRODUCT FORM FOR THE INVERSE IN THE SIMPLEX METHOD , 1954 .

[3]  H. Markowitz.  The Elimination form of the Inverse and its Application to Linear Programming , 1957 .

[4]  George B. Dantzig,et al.  Decomposition Principle for Linear Programs , 1960 .

[5]  R. E. Griffith,et al.  A Nonlinear Programming Technique for the Optimization of Continuous Processing Systems , 1961 .

[6]  G. Dantzig.  23. A Decomposition Principle for Linear Programs , 1963 .

[7]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[8]  J. H. Wilkinson.  The algebraic eigenvalue problem , 1966 .

[9]  M. J. D. Powell,et al.  Nonlinear Programming—Sequential Unconstrained Minimization Techniques , 1969 .

[10]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[11]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[12]  Gene H. Golub,et al.  The simplex method of linear programming using LU decomposition , 1969, Commun. ACM.

[13]  Lenhart K. Schubert.  Modification of a quasi-Newton method for nonlinear equations with a sparse Jacobian , 1970 .

[14]  G. Golub,et al.  Numerical techniques in mathematical programming , 1970 .

[15]  Eli Hellerman,et al.  Reinversion with the preassigned pivot procedure , 1971, Math. Program..

[16]  R. Bartels.  A stabilization of the simplex method , 1971 .

[17]  John A. Tomlin,et al.  Updated triangular factors of the basis to maintain sparsity in the product form simplex method , 1972, Math. Program..

[18]  Eli Hellerman,et al.  The Partitioned Preassigned Pivot Procedure (P4) , 1972 .

[19]  Fred G. Gustavson,et al.  Some Basic Techniques for Solving Sparse Systems of Linear Equations , 1972 .

[20]  S. M. Robinson,et al.  A quadratically-convergent algorithm for general nonlinear programming problems , 1972, Math. Program..

[21]  S. Vajda,et al.  Numerical Methods for Non-Linear Optimization , 1973 .

[22]  J. Reid.  Large Sparse Sets of Linear Equations , 1973 .

[23]  Philip E. Gill,et al.  Newton-type methods for unconstrained and linearly constrained optimization , 1974, Math. Program..

[24]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[25]  O Axelsson,et al.  On preconditioning and convergence acceleration in sparse matrix problems , 1974 .

[26]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[27]  R. Cottle.  On manifestations of the Schur complement , 1975 .

[28]  J. A. Tomlin.  An accuracy test for updating triangular factors , 1975 .

[29]  Michael A. Saunders,et al.  A FAST, STABLE IMPLEMENTATION OF THE SIMPLEX METHOD USING BARTELS-GOLUB UPDATING , 1976 .

[30]  Alexander Meeraus,et al.  Matrix augmentation and partitioning in the updating of the basis inverse , 1977, Math. Program..

[31]  Harvey J. Greenberg,et al.  Design and implementation of optimization software , 1977, Math. Program..

[32]  P. Toint.  On sparse and symmetric matrix updating subject to a linear equation , 1977 .

[33]  P. Toint.  Some numerical results using a sparse matrix updating formula in unconstrained optimization , 1978 .

[34]  E. M. L. Beale.  Nonlinear Programming Using a General Mathematical Programming System , 1978 .

[35]  R. Schnabel,et al.  Least Change Secant Updates for Quasi-Newton Methods , 1978 .

[36]  J. B. Rosen.  TWO-PHASE ALGORITHM FOR NONLINEAR CONSTRAINT PROBLEMS11This research was supported in part by the National Science Foundation grant MCS 76-23311. Support by the Systems Optimization Lab, Department of Operations Research, Stanford University, during the author's sabbatical leave, is also gratefully , 1978 .

[37]  Michael A. Saunders,et al.  Large-scale linearly constrained optimization , 1978, Math. Program..

[38]  P. Toint.  On the Superlinear Convergence of an Algorithm for Solving a Sparse Minimization Problem , 1979 .

[39]  P. Gill,et al.  Conjugate-Gradient Methods for Large-Scale Nonlinear Optimization. , 1979 .

[40]  Arthur Djang.  Algorithmic equivalence in quadratic programming , 1979 .

[41]  M. Powell,et al.  On the Estimation of Sparse Hessian Matrices , 1979 .

[42]  T. M. Williams,et al.  Practical Methods of Optimization. Vol. 1: Unconstrained Optimization , 1980 .

[43]  A. George,et al.  Solution of sparse linear least squares problems using givens rotations , 1980 .

[44]  Alexander Meeraus,et al.  Matrix augmentation and structure preservation in linearly constrained control problems , 1980, Math. Program..

[45]  Magnus R. Hestenes,et al.  Conjugate Direction Methods in Optimization , 1980 .

[46]  D. Shanno.  On variable-metric methods for sparse Hessians , 1980 .

[47]  P. Gill,et al.  QP-Based Methods for Large-Scale Nonlinearly Constrained Optimization. , 1981 .

[48]  Nesa L'abbe Wu,et al.  Linear programming and extensions , 1981 .

[49]  G. Dantzig,et al.  Large-scale linear programming : proceedings of a IIASA workshop, 2-6 June 1980 , 1981 .

[50]  Gerald G. Brown,et al.  Automatic Identification of Embedded Structure in Large-Scale Optimization Models , 1981 .

[51]  Philip E. Gill,et al.  Practical optimization , 1981 .

[52]  M. J. D. Powell,et al.  A note on quasi-newton formulae for sparse second derivative matrices , 1981, Math. Program..

[53]  Dianne P. O'Leary,et al.  A discrete Newton algorithm for minimizing a function of many variables , 1982, Math. Program..

[54]  D. Sorensen.  Collinear scaling and sequential estimation in sparse optimization algorithms , 1982 .

[55]  J. Dennis,et al.  Direct secant updates of matrix factorizations , 1982 .

[56]  Margaret H. Wright,et al.  Computation of the search direction in constrained optimization algorithms , 1982 .

[57]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[58]  Michael A. Saunders,et al.  A projected Lagrangian algorithm and its implementation for sparse nonlinear constraints , 1982 .

[59]  Robert Fourer,et al.  Solving staircase linear programs by the simplex method, 1: Inversion , 1982, Math. Program..

[60]  P. Gill,et al.  Range-Space Methods for Convex Quadratic Programming. , 1982 .

[61]  John K. Reid,et al.  A sparsity-exploiting variant of the Bartels—Golub decomposition for linear programming bases , 1982, Math. Program..

[62]  S. Nash.  Truncated-Newton methods , 1982 .

[63]  L. Lasdon,et al.  Nonlinear Optimization by Successive Linear Programming , 1982 .

[64]  M. Heath.  Some Extensions of an Algorithm for Sparse Linear Least Squares Problems , 1982 .

[65]  J. J. Moré,et al.  Newton's Method , 1982 .

[66]  M. J. D. Powell,et al.  Variable Metric Methods for Constrained Optimization , 1982, ISMP.

[67]  S. Thomas McCormick,et al.  Optimal approximation of sparse hessians and its equivalence to a graph coloring problem , 1983, Math. Program..

[68]  J. J. Moré,et al.  Estimation of sparse jacobian matrices and graph coloring problems , 1983 .

[69]  John K. Reid,et al.  The Multifrontal Solution of Indefinite Sparse Symmetric Linear , 1983, TOMS.

[70]  Mukund N. Thapa,et al.  Optimization of unconstrained functions with sparse Hessian matrices—Quasi-Newton methods , 1983, Math. Program..

[71]  John D. Ramsdell,et al.  Estimation of Sparse Jacobian Matrices , 1983 .

[72]  Michael T. Heath,et al.  Solution of Sparse Underdetermined Systems of Linear Equations , 1984 .

[73]  I. Duff.  Direct Methods for Solving Sparse Systems of Linear Equations , 1984 .

[74]  Thomas F. Coleman,et al.  Software for estimating sparse Jacobian matrices , 1984, ACM Trans. Math. Softw..

[75]  Thomas F. Coleman,et al.  Estimation of sparse hessian matrices and graph coloring problems , 1982, Math. Program..