Convergence rates for Kaczmarz-type algorithms

In this paper, we make a theoretical analysis of the convergence rates of Kaczmarz and extended Kaczmarz projection algorithms for some of the most practically used control sequences. We first prove an at least linear convergence rate for the Kaczmarz-Tanabe and its extended version methods (the one in which a complete set of projections using row/column indices is performed in each iteration). Then, we apply the main ideas of this analysis in establishing an at least sublinear, respectively, linear convergence rate for the Kaczmarz algorithm with almost cyclic and the remotest set control strategies, and their extended versions, respectively. These results complete the existing ones related to the random selection procedures.

[1]  C. Popa.  Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems , 1998 .

[2]  S. Kaczmarz.  Approximate solution of systems of linear equations , 1993 .

[3]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[4]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[5]  Y. Censor,et al.  Parallel Optimization: Theory, Algorithms, and Applications , 1997 .

[6]  Stefania Petra,et al.  Single projection Kaczmarz extended algorithms , 2015, Numerical Algorithms.

[7]  Deanna Needell,et al.  Convergence Properties of the Randomized Extended Gauss-Seidel and Kaczmarz Methods , 2015, SIAM J. Matrix Anal. Appl..

[8]  Y. Censor,et al.  A Note on the Behavior of the Randomized Kaczmarz Algorithm of Strohmer and Vershynin , 2009, The journal of fourier analysis and applications.

[9]  Thomas B. Schön,et al.  On the Exponential Convergence of the Kaczmarz Algorithm , 2015, IEEE Signal Processing Letters.

[10]  Andrzej Cegielski,et al.  Projection methods: an annotated bibliography of books and reviews , 2014, 1406.6143.

[11]  Peter Richtárik,et al.  A new perspective on randomized gossip algorithms , 2016, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[12]  Walter Gautschi,et al.  Numerical Analysis , 1978, Mathemagics: A Magical Journey Through Advanced Mathematics.

[13]  Wei Chen,et al.  Finite Convergence of a Subgradient Projections Method with Expanding Controls , 2011 .

[14]  Y. Censor.  Row-Action Methods for Huge and Sparse Systems and Their Applications , 1981 .

[15]  R. Vershynin,et al.  Comments on the Randomized Kaczmarz Method , 2009 .

[16]  Jan Mandel.  Convergence of the cyclical relaxation method for linear inequalities , 1984, Math. Program..

[17]  P. Oswald,et al.  Convergence analysis for Kaczmarz-type methods in a Hilbert space framework , 2015 .

[18]  Mark W. Schmidt,et al.  Convergence Rates for Greedy Kaczmarz Algorithms, and Randomized Kaczmarz Rules Using the Orthogonality Graph , 2016, UAI.

[19]  P. L. Combettes,et al.  Hilbertian convex feasibility problem: Convergence of projection methods , 1997 .

[20]  Peter Richtárik,et al.  Randomized Iterative Methods for Linear Systems , 2015, SIAM J. Matrix Anal. Appl..

[21]  Constantin Popa,et al.  Least-squares solution of overdetermined inconsistent linear systems using kaczmarz's relaxation , 1995, Int. J. Comput. Math..

[22]  Constantin Popa,et al.  Projection Algorithms - Classical Results and Developments , 2012 .

[23]  Andrzej Stachurski,et al.  Parallel Optimization: Theory, Algorithms and Applications , 2000, Parallel Distributed Comput. Pract..

[24]  On Remotest set and Random controls in Kaczmarz algorithm , 2017, 1704.04660.

[25]  K. Tanabe.  Projection method for solving a singular system of linear equations and its applications , 1971 .

[26]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[27]  Rainer Ansorge,et al.  Connections between the Cimmino-method and the Kaczmarz-method for the solution of singular and regular systems of equations , 1984, Computing.

[28]  G. Herman,et al.  Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and x-ray photography. , 1970, Journal of theoretical biology.

[29]  R. Sznajder.  Kaczmarz algorithm revisited , 2016 .