Convergence Rates for Greedy Kaczmarz Algorithms, and Randomized Kaczmarz Rules Using the Orthogonality Graph

The Kaczmarz method is an iterative algorithm for solving systems of linear equalities and inequalities, that iteratively projects onto these constraints. Recently, Strohmer and Vershynin [J. Fourier Anal. Appl., 15(2):262-278, 2009] gave a non-asymptotic convergence rate analysis for this algorithm, spurring numerous extensions and generalizations of the Kaczmarz method. Rather than the randomized selection rule analyzed in that work, in this paper we instead discuss greedy and approximate greedy selection rules. We show that in some applications the computational costs of greedy and random selection are comparable, and that in many cases greedy selection rules give faster convergence rates than random selection rules. Further, we give the first multi-step analysis of Kaczmarz methods for a particular greedy rule, and propose a provably-faster randomized selection rule for matrices with many pairwise-orthogonal rows.

[1]  Deanna Needell,et al.  Convergence Properties of the Randomized Extended Gauss-Seidel and Kaczmarz Methods , 2015, SIAM J. Matrix Anal. Appl..

[2]  Peter Richtárik,et al.  Randomized Iterative Methods for Linear Systems , 2015, SIAM J. Matrix Anal. Appl..

[3]  Y. Censor.  Row-Action Methods for Huge and Sparse Systems and Their Applications , 1981 .

[4]  Hans G. Feichtinger,et al.  New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling , 1992, Other Conferences.

[5]  Stephen J. Wright,et al.  An accelerated randomized Kaczmarz algorithm , 2013, Math. Comput..

[6]  A. Galántai.  On the rate of convergence of the alternating projection method in finite dimensional spaces , 2005 .

[7]  Gabor T. Herman,et al.  Algebraic reconstruction techniques can be made computationally efficient [positron emission tomography application] , 1993, IEEE Trans. Medical Imaging.

[8]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[9]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[10]  Mark W. Schmidt,et al.  Minimizing finite sums with the stochastic average gradient , 2013, Mathematical Programming.

[11]  Mark W. Schmidt,et al.  Coordinate Descent Converges Faster with the Gauss-Southwell Rule Than Random Selection , 2015, ICML.

[12]  Y. Censor,et al.  Strong underrelaxation in Kaczmarz's method for inconsistent systems , 1983 .

[13]  A. Sluis.  Condition numbers and equilibration of matrices , 1969 .

[14]  Hein Hundal,et al.  The Rate of Convergence for the Method of Alternating Projections, II , 1997 .

[15]  Behrooz Sepehry.  Finding a Maximum Weight Sequence with Dependency Constraints , 2016 .

[16]  P. Oswald,et al.  Convergence analysis for Kaczmarz-type methods in a Hilbert space framework , 2015 .

[17]  Deanna Needell,et al.  Stochastic gradient descent and the randomized Kaczmarz algorithm , 2013, ArXiv.

[18]  Adrian S. Lewis,et al.  Randomized Methods for Linear Constraints: Convergence Rates and Conditioning , 2008, Math. Oper. Res..

[19]  K. Tanabe.  Projection method for solving a singular system of linear equations and its applications , 1971 .

[20]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[21]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[22]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[23]  Yonina C. Eldar,et al.  Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma , 2010, Numerical Algorithms.

[24]  Stephen J. Wright,et al.  An Asynchronous Parallel Randomized Kaczmarz Algorithm , 2014, ArXiv.

[25]  M. Hanke,et al.  On the acceleration of Kaczmarz's method for inconsistent linear systems , 1990 .

[26]  A. Hoffman.  On approximate solutions of systems of linear inequalities , 1952 .

[27]  Nicolas Le Roux,et al.  Label Propagation and Quadratic Criterion , 2006, Semi-Supervised Learning.

[28]  Yin Tat Lee,et al.  Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[29]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[30]  P. Oswald,et al.  Greedy and Randomized Versions of the Multiplicative Schwarz Method , 2012 .

[31]  Albert B Novikoff,et al.  ON CONVERGENCE PROOFS FOR PERCEPTRONS , 1963 .

[32]  Lizhi Cheng,et al.  An accelerated randomized Kaczmarz method via low-rank approximation , 2015, Int. J. Comput. Math..

[33]  Y. Censor,et al.  A Note on the Behavior of the Randomized Kaczmarz Algorithm of Strohmer and Vershynin , 2009, The journal of fourier analysis and applications.

[34]  T. Whitney,et al.  Two Algorithms Related to the Method of Steepest Descent , 1967 .

[35]  Gabor T. Herman.  Algebraic Reconstruction Techniques , 2009 .

[36]  F. Deutsch.  Rate of Convergence of the Method of Alternating Projections , 1984 .

[37]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[38]  Stephen J. Wright.  Coordinate descent algorithms , 2015, Mathematical Programming.

[39]  Mark W. Schmidt,et al.  Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition , 2016, ECML/PKDD.

[40]  Deanna Needell,et al.  Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm , 2013, Mathematical Programming.