A Randomized Solver for Linear Systems with Exponential Convergence

The Kaczmarz method for solving linear systems of equations Ax=b is an iterative algorithm that has found many applications ranging from computer tomography to digital signal processing. Despite the popularity of this method, useful theoretical estimates for its rate of convergence are still scarce. We introduce a randomized version of the Kaczmarz method for overdetermined linear systems and we prove that it converges with expected exponential rate. Furthermore, this is the first solver whose rate does not depend on the number of equations in the system. The solver does not even need to know the whole system, but only its small random part. It thus outperforms all previously known methods on extremely overdetermined systems. Even for moderately overdetermined systems, numerical simulations reveal that our algorithm can converge faster than the celebrated conjugate gradient algorithm.

[1]  C. Berry.  A pharmacy coordinated unit dose dispensing and drug administration system. Description of the system. , 1970, American journal of hospital pharmacy.

[2]  Gabor T. Herman,et al.  Image Reconstruction From Projections , 1975, Real Time Imaging.

[3]  Gabor T. Herman,et al.  Image reconstruction from projections : the fundamentals of computerized tomography , 1980 .

[4]  S. Geman.  A Limit Theorem for the Norm of Random Matrices , 1980 .

[5]  Gene H. Golub,et al.  Matrix computations , 1983 .

[6]  F. Deutsch.  Rate of Convergence of the Method of Alternating Projections , 1984 .

[7]  H. V. D. Vorst,et al.  The rate of convergence of Conjugate Gradients , 1986 .

[8]  F. Natterer.  The Mathematics of Computerized Tomography , 1986 .

[9]  Henry Stark,et al.  Image recovery: Theory and application , 1987 .

[10]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[11]  Hans G. Feichtinger,et al.  New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling , 1992, Other Conferences.

[12]  Gabor T. Herman,et al.  Algebraic reconstruction techniques can be made computationally efficient [positron emission tomography application] , 1993, IEEE Trans. Medical Imaging.

[13]  G. Hounsfield.  Computerized transverse axial scanning (tomography): Part I. Description of system. 1973. , 1973, The British journal of radiology.

[14]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[15]  Hein Hundal,et al.  The Rate of Convergence for the Method of Alternating Projections, II , 1997 .

[16]  Alan M. Frieze,et al.  Fast monte-carlo algorithms for finding low-rank approximations , 2004, JACM.

[17]  A. Galántai.  On the rate of convergence of the alternating projection method in finite dimensional spaces , 2005 .

[18]  Mark Rudelson,et al.  Sampling from large matrices: An approach through geometric functional analysis , 2005, JACM.