On the Exponential Convergence of the Kaczmarz Algorithm

The Kaczmarz algorithm (KA) is a popular method for solving a system of linear equations. In this note we derive a new exponential convergence result for the KA. The key allowing us to establish the new result is to rewrite the KA in such a way that its solution path can be interpreted as the output from a particular dynamical system. The asymptotic stability results of the corresponding dynamical system can then be leveraged to prove exponential convergence of the KA. The new bound is also compared to existing bounds.

[1]  Yonina C. Eldar,et al.  Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma , 2010, Numerical Algorithms.

[2]  Jan Mandel.  Convergence of the cyclical relaxation method for linear inequalities , 1984, Math. Program..

[3]  L. Ljung,et al.  Exponential stability of general tracking algorithms , 1995, IEEE Trans. Autom. Control..

[4]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[5]  D. Manjunath,et al.  A stochastic Kaczmarz algorithm for network tomography , 2012, Autom..

[6]  P. Oswald,et al.  Convergence analysis for Kaczmarz-type methods in a Hilbert space framework , 2015 .

[7]  R. Vershynin,et al.  Comments on the Randomized Kaczmarz Method , 2009 .

[8]  Kristiaan Pelckmans,et al.  On the Randomized Kaczmarz Algorithm , 2014, IEEE Signal Processing Letters.

[9]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[10]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[11]  A. Hmamed,et al.  A matrix inequality , 1989 .

[12]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[13]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[14]  A. Galántai.  On the rate of convergence of the alternating projection method in finite dimensional spaces , 2005 .

[15]  Y. Censor.  Row-Action Methods for Huge and Sparse Systems and Their Applications , 1981 .

[16]  F. Natterer.  The Mathematics of Computerized Tomography , 1986 .

[17]  Xuemei Chen,et al.  Almost Sure Convergence of the Kaczmarz Algorithm with Random Measurements , 2012 .

[18]  A. Lent,et al.  Iterative algorithms for large partitioned linear systems, with applications to image reconstruction , 1981 .