Differentiating the Method of Conjugate Gradients

The method of conjugate gradients (CG) is widely used for the iterative solution of large sparse systems of equations $Ax=b$, where $A\in\Re^{n\times n}$ is symmetric positive definite. Let $x_k$ denote the $k$th iterate of CG. This is a nonlinear differentiable function of $b$. In this paper we obtain expressions for $J_k$, the Jacobian matrix of $x_k$ with respect to $b$. We use these expressions to obtain bounds on $\|J_k\|_2$, the spectral norm condition number of $x_k$, and discuss algorithms to compute or estimate $J_kv$ and $J_k^Tv$ for a given vector $v$.

[1]  C. Paige.  Error Analysis of the Lanczos Algorithm for Tridiagonalizing a Symmetric Matrix , 1976 .

[2]  R. Winther.  Some Superlinear Convergence Results for the Conjugate Gradient Method , 1980 .

[3]  Nicholas J. Higham,et al.  Functions of matrices - theory and computation , 2008 .

[4]  R. Gelaro,et al.  Observation Sensitivity Calculations Using the Adjoint of the Gridpoint Statistical Interpolation (GSI) Analysis System , 2008 .

[5]  Anne Greenbaum,et al.  Iterative methods for solving linear systems , 1997, Frontiers in applied mathematics.

[6]  Christopher C. Paige,et al.  A Useful Form of Unitary Matrix Obtained from Any Sequence of Unit 2-Norm n-Vectors , 2009, SIAM J. Matrix Anal. Appl..

[7]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[8]  Sergey V. Kuznetsov.  Perturbation bounds of the krylov bases and associated hessenberg forms , 1997 .

[9]  B. Fischer.  Polynomial Based Iteration Methods for Symmetric Linear Systems , 1996 .

[10]  G. Meurant,et al.  The Lanczos and conjugate gradient algorithms in finite precision arithmetic , 2006, Acta Numerica.

[11]  G. Meurant.  Computer Solution of Large Linear Systems , 1999 .

[12]  Christopher C. Paige,et al.  An Augmented Stability Result for the Lanczos Hermitian Matrix Tridiagonalization Process , 2010, SIAM J. Matrix Anal. Appl..

[13]  Henk A. van der Vorst,et al.  Approximate solutions and eigenvalue bounds from Krylov subspaces , 1995, Numer. Linear Algebra Appl..

[14]  G. Meurant.  The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations , 2006 .

[15]  A. Wathen,et al.  Chebyshev semi-iteration in preconditioning for problems including the mass matrix. , 2008 .

[16]  Z. Strakos,et al.  On the real convergence rate of the conjugate gradient method , 1991 .

[17]  A. Griewank,et al.  Automatic differentiation of algorithms : theory, implementation, and application , 1994 .

[18]  Anne Greenbaum,et al.  Predicting the Behavior of Finite Precision Lanczos and Conjugate Gradient Computations , 2015, SIAM J. Matrix Anal. Appl..

[19]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[20]  Alston S. Householder,et al.  The Theory of Matrices in Numerical Analysis , 1964 .

[21]  A. Greenbaum.  Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences , 1989 .

[22]  Lars Eldén,et al.  Partial least-squares vs. Lanczos bidiagonalization - I: analysis of a projection method for multiple regression , 2004, Comput. Stat. Data Anal..

[23]  Z. Strakos,et al.  Krylov Subspace Methods: Principles and Analysis , 2012 .

[24]  A. Greenbaum.  Comparison of splittings used with the conjugate gradient algorithm , 1979 .

[25]  J. Magnus,et al.  Matrix Differential Calculus with Applications in Statistics and Econometrics , 1991 .

[26]  S. Godunov,et al.  Condition number of the Krylov bases and subspaces , 1996 .

[27]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .