Solution of Large Linear Systems of Equations by Conjugate Gradient Type Methods

In 1952, Hestenes and Stiefel introduced the conjugate gradient algorithm in their landmark paper [27] as an algorithm for solving linear equation Ax = b with A as positive definite n × n-matrix (see the book [26] of Hestenes for a broad exposition). The algorithm fascinated numerical analysts since then for various reasons: The cg-algorithm combines features of direct and iterative methods which attracted the attention in the early years: It generates a sequence x i of vectors approximating the solution $$\overline x$$ in a defined way like other iterative methods, but like direct methods, terminating with the exact solution after at most n steps, at least in theory. Many expectations were disappointed, when it was found out that due to roundoff the n-step termination property does not hold in practice. However, viewed as an iterative method, the cg-algorithm has very attractive features. Its application for the iterative solution of large sparse systems has been discussed very early [11] by Stiefel and his coworkers. Like other iterative methods, it essentially requires only the formation of one matrix-vector product A·x per iteration, so that the iterations are inexpensive even for large matrices A, if they are sparse. The iterative aspect of the method has been particularly emphasized since the work of Reid [38].

[1]  Magnus R. Hestenes,et al.  Conjugate Direction Methods in Optimization , 1980 .

[2]  O. Axelsson,et al.  A generalized conjugate direction method and its application on a singular perturbation problem , 1980 .

[3]  Philip E. Gill,et al.  Newton-type methods for unconstrained and linearly constrained optimization , 1974, Math. Program..

[4]  Alston S. Householder,et al.  The Theory of Matrices in Numerical Analysis , 1964 .

[5]  E. J. Craig.  The N‐Step Iteration Procedures , 1955 .

[6]  E. Stiefel,et al.  Refined Iterative Methods for Computation of the Solution and the Eigenvalues of Self-Adjoint Boundary Value Problems , 1959 .

[7]  P. Toint,et al.  Partitioned variable metric updates for large structured optimization problems , 1982 .

[8]  J. Gillis,et al.  Matrix Iterative Analysis , 1961 .

[9]  A. Greenbaum.  Comparison of splittings used with the conjugate gradient algorithm , 1979 .

[10]  I. Gustafsson.  Stability and rate of convergence of modified incomplete Cholesky factorization methods , 1979 .

[11]  Kang C. Jea,et al.  Generalized conjugate-gradient acceleration of nonsymmetrizable iterative methods , 1980 .

[12]  O Axelsson,et al.  On preconditioning and convergence acceleration in sparse matrix problems , 1974 .

[13]  R. Varga,et al.  Incomplete Factorizations of Matrices and Connections with H-Matrices , 1980 .

[14]  O. Axelsson.  Conjugate gradient type methods for unsymmetric and inconsistent systems of linear equations , 1980 .

[15]  Dianne P. O'Leary,et al.  A discrete Newton algorithm for minimizing a function of many variables , 1982, Math. Program..

[16]  H. Simon.  The lanczos algorithm for solving symmetric linear systems , 1982 .

[17]  R. Chandra.  Conjugate gradient methods for partial differential equations. , 1978 .

[18]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[19]  Franklin T. Luk,et al.  ON THE EQUIVALENCE OF CERTAIN ITERATIVE ACCELERATION METHODS , 1980 .

[20]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[21]  J. Reid.  Large Sparse Sets of Linear Equations , 1973 .

[22]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[23]  T. Manteuffel.  An incomplete factorization technique for positive definite linear systems , 1980 .

[24]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[25]  R. Glowinski,et al.  Computing Methods in Applied Sciences and Engineering , 1974 .

[26]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[27]  I. Gustafsson,et al.  A Modified Upwind Scheme for Convective Transport Equations and the Use of a Conjugate Gradient Method for the Solution of Non-Symmetric Systems of Equations , 1977 .

[28]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[29]  D. Luenberger.  Hyperbolic Pairs in the Method of Conjugate Gradients , 1969 .

[30]  P. Toint,et al.  Local convergence analysis for partitioned quasi-Newton updates , 1982 .

[31]  Louis A. Hageman,et al.  Iterative Solution of Large Linear Systems. , 1971 .

[32]  S. Nash.  Truncated-Newton methods , 1982 .

[33]  V. Fridman,et al.  The method of minimum iterations with minimum errors for a system of linear algebraic equations with a symmetrical matrix , 1963 .

[34]  D. O’Leary.  A generalized conjugate gradient algorithm for solving a class of quadratic programming problems , 1977 .

[35]  D. Faddeev,et al.  Computational methods of linear algebra , 1981 .

[36]  O. Axelsson.  Solution of linear systems of equations: Iterative methods , 1977 .

[37]  R. Fletcher.  Conjugate gradient methods for indefinite systems , 1976 .

[38]  J. Meijerink,et al.  An iterative solution method for linear systems of which the coefficient matrix is a symmetric -matrix , 1977 .

[39]  O. Widlund.  A Lanczos Method for a Class of Nonsymmetric Systems of Linear Equations , 1978 .

[40]  I. Gustafsson.  A class of first order factorization methods , 1978 .