Stochastic iterative projection methods for large linear systems

Abstract We suggest a randomized version of the projection methods belonging to the class of a “row-action” methods which work well both for systems with quadratic nonsingular matrices and for overdetermined systems. These methods belong to a type known as Projection on Convex Sets methods. Here we present a method beyond the conventional Markov chain based Neumann–Ulam scheme. The main idea is in a random choice of blocks of rows in the projection method so that in average, the convergence is improved compared to the conventional periodic choice of the rows. We suggest an acceleration of the row projection method by using the Johnson–Lindenstrauss (J–L) theorem to find, among the randomly chosen rows, in a sense an optimal row. We extend this randomized method for solving linear systems coupled with systems of linear inequalities. Applied to finite-difference approximations of boundary value problems, the method appears to be an extremely efficient Random Walk algorithm whose convergence is exponential, and the cost does not depend on the dimension of the matrix. In addition, the algorithm calculates the solution in all grid points, and is easily parallelizable.

[1]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[2]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[3]  Karl K. Sabelfeld,et al.  Sparsified Randomization Algorithms for large systems of linear equations and a new version of the Random Walk on Boundary method , 2009, Monte Carlo Methods Appl..

[4]  Santosh S. Vempala,et al.  The Random Projection Method , 2005, DIMACS Series in Discrete Mathematics and Theoretical Computer Science.

[5]  Bernard Chazelle,et al.  The Fast Johnson--Lindenstrauss Transform and Approximate Nearest Neighbors , 2009, SIAM J. Comput..

[6]  Karl K. Sabelfeld.  Monte Carlo Methods in Boundary Value Problems. , 1991 .

[7]  A. J. Walker.  New fast method for generating discrete random numbers with arbitrary frequency distributions , 1974 .

[8]  Mark Tygert,et al.  A Randomized Algorithm for Principal Component Analysis , 2008, SIAM J. Matrix Anal. Appl..

[9]  Dimitris Achlioptas,et al.  Fast computation of low-rank matrix approximations , 2007, JACM.

[10]  Petros Drineas,et al.  Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication , 2006, SIAM J. Comput..

[11]  M. Kobayashi,et al.  Estimation of singular values of very large matrices using random sampling , 2001 .

[12]  Erich Kaltofen,et al.  On randomized Lanczos algorithms , 1997, ISSAC.

[13]  K. Sabelfeld,et al.  Random Walks on Boundary for Solving PDEs , 1994 .

[14]  Tamás Sarlós,et al.  Improved Approximation Algorithms for Large Matrices via Random Projections , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[15]  V. Rokhlin,et al.  A fast randomized algorithm for the approximation of matrices ✩ , 2007 .

[16]  Alan M. Frieze,et al.  Fast monte-carlo algorithms for finding low-rank approximations , 2004, JACM.

[17]  K. Tanabe.  Projection method for solving a singular system of linear equations and its applications , 1971 .

[18]  Kenneth Ward Church,et al.  Very sparse random projections , 2006, KDD '06.