An Asynchronous Parallel Randomized Kaczmarz Algorithm

We describe an asynchronous parallel variant of the randomized Kaczmarz (RK) algorithm for solving the linear system Ax = b. The analysis shows linear convergence and indicates that nearly linear speedup can be expected if the number of processors is bounded by a multiple of the number of rows in A.

[1]  A. Hoffman.  On approximate solutions of systems of linear inequalities , 1952 .

[2]  Gabor T. Herman,et al.  Image reconstruction from projections : the fundamentals of computerized tomography , 1980 .

[3]  Y. Censor.  Row-Action Methods for Huge and Sparse Systems and Their Applications , 1981 .

[4]  A. Lent,et al.  Iterative algorithms for large partitioned linear systems, with applications to image reconstruction , 1981 .

[5]  Y. Censor,et al.  Block-iterative projection methods for parallel computation of solutions to convex feasibility problems , 1989 .

[6]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[7]  L. Elsner,et al.  On the convergence of asynchronous paracontractions with application to tomographic reconstruction from incomplete data , 1990 .

[8]  Michael C. Ferris,et al.  Parallel Variable Distribution , 1994, SIAM J. Optim..

[9]  C. Popa.  Characterization of the solutions set of least-squares problems by an extension of Kaczmarz's projec , 1997 .

[10]  C. Popa.  Characterization of the solutions set of inconsistent least-squares problems by an extended Kaczmarz algorithm , 1999 .

[11]  Yair Censor,et al.  Component averaging: An efficient iterative parallel algorithm for large and sparse unstructured problems , 2001, Parallel Comput..

[12]  A. Galántai.  On the rate of convergence of the alternating projection method in finite dimensional spaces , 2005 .

[13]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[14]  T. Nikazad.  Algebraic Reconstruction Methods , 2008 .

[15]  Gabor T. Herman,et al.  Fundamentals of Computerized Tomography: Image Reconstruction from Projections , 2009, Advances in Pattern Recognition.

[16]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[17]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[18]  T. Elfving,et al.  Properties of a class of block-iterative methods , 2009 .

[19]  Adrian S. Lewis,et al.  Randomized Methods for Linear Constraints: Convergence Rates and Conditioning , 2008, Math. Oper. Res..

[20]  Stephen J. Wright,et al.  Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent , 2011, NIPS.

[21]  Yonina C. Eldar,et al.  Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma , 2010, Numerical Algorithms.

[22]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[23]  Haim Avron,et al.  A Randomized Asynchronous Linear Solver with Provable Convergence Rate , 2013, ArXiv.

[24]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[25]  Stephen J. Wright,et al.  An asynchronous parallel stochastic coordinate descent algorithm , 2013, J. Mach. Learn. Res..

[26]  Stephen J. Wright,et al.  An accelerated randomized Kaczmarz algorithm , 2013, Math. Comput..

[27]  Peter Richtárik,et al.  Parallel coordinate descent methods for big data optimization , 2012, Mathematical Programming.