Convergence Rates for Greedy Kaczmarz Algorithms, and Randomized Kaczmarz Rules Using the Orthogonality Graph

The Kaczmarz method is an iterative algorithm for solving systems of linear equalities and inequalities, that iteratively projects onto these constraints. Recently, Strohmer and Vershynin [J. Fourier Anal. Appl., 15(2):262-278, 2009] gave a non-asymptotic convergence rate analysis for this algorithm, spurring numerous extensions and generalizations of the Kaczmarz method. Rather than the randomized selection rule analyzed in that work, in this paper we instead discuss greedy and approximate greedy selection rules. We show that in some applications the computational costs of greedy and random selection are comparable, and that in many cases greedy selection rules give faster convergence rates than random selection rules. Further, we give the first multi-step analysis of Kaczmarz methods for a particular greedy rule, and propose a provably-faster randomized selection rule for matrices with many pairwise-orthogonal rows.

[1]  Y. Censor,et al.  A Note on the Behavior of the Randomized Kaczmarz Algorithm of Strohmer and Vershynin , 2009, The journal of fourier analysis and applications.

[2]  Y. Censor,et al.  Strong underrelaxation in Kaczmarz's method for inconsistent systems , 1983 .

[3]  Deanna Needell,et al.  Convergence Properties of the Randomized Extended Gauss-Seidel and Kaczmarz Methods , 2015, SIAM J. Matrix Anal. Appl..

[4]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[5]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[6]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[7]  Albert B Novikoff,et al.  ON CONVERGENCE PROOFS FOR PERCEPTRONS , 1963 .

[8]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[9]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[10]  Gabor T. Herman,et al.  Algebraic reconstruction techniques can be made computationally efficient [positron emission tomography application] , 1993, IEEE Trans. Medical Imaging.

[11]  K. Tanabe.  Projection method for solving a singular system of linear equations and its applications , 1971 .

[12]  Hein Hundal,et al.  The Rate of Convergence for the Method of Alternating Projections, II , 1997 .

[13]  Peter Richtárik,et al.  Randomized Iterative Methods for Linear Systems , 2015, SIAM J. Matrix Anal. Appl..

[14]  C. Badea,et al.  The rate of convergence in the method of alternating projections , 2010, 1006.2047.

[15]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[16]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[17]  T. Whitney,et al.  Two Algorithms Related to the Method of Steepest Descent , 1967 .

[18]  Alexander Zien,et al.  Label Propagation and Quadratic Criterion , 2006 .

[19]  A. Galántai.  On the rate of convergence of the alternating projection method in finite dimensional spaces , 2005 .

[20]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[21]  Hans G. Feichtinger,et al.  New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling , 1992, Other Conferences.

[22]  Y. Censor.  Row-Action Methods for Huge and Sparse Systems and Their Applications , 1981 .

[23]  Adrian S. Lewis,et al.  Randomized Methods for Linear Constraints: Convergence Rates and Conditioning , 2008, Math. Oper. Res..

[24]  P. Oswald,et al.  Convergence analysis for Kaczmarz-type methods in a Hilbert space framework , 2015 .

[25]  Deanna Needell,et al.  Stochastic gradient descent and the randomized Kaczmarz algorithm , 2013, ArXiv.

[26]  M. Hanke,et al.  On the acceleration of Kaczmarz's method for inconsistent linear systems , 1990 .

[27]  Yin Tat Lee,et al.  Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[28]  Mark W. Schmidt,et al.  Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection , 2015, ICML.

[29]  A. Hoffman.  On approximate solutions of systems of linear inequalities , 1952 .

[30]  Yonina C. Eldar,et al.  Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma , 2010, Numerical Algorithms.

[31]  P. Oswald,et al.  Greedy and Randomized Versions of the Multiplicative Schwarz Method , 2012 .

[32]  G. Herman,et al.  Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and x-ray photography. , 1970, Journal of theoretical biology.