Stochastic Algorithms in Linear Algebra - beyond the Markov Chains and von Neumann - Ulam Scheme

Sparsified Randomization Monte Carlo (SRMC) algorithms for solving systems of linear algebraic equations introduced in our previous paper [34] are discussed here in a broader context. In particular, I present new randomized solvers for large systems of linear equations, randomized singular value (SVD) decomposition for large matrices and their use for solving inverse problems, and stochastic simulation of random fields. Stochastic projection methods, which I call here "random row action" algorithms, are extended to problems which involve systems of equations and constrains in the form of systems of linear inequalities.

[1]  Sean McKee,et al.  Monte Carlo Methods for Applied Scientists , 2005 .

[2]  Jitendra Malik,et al.  Spectral Partitioning with Inde nite Kernels using the Nystr om Extension , 2002 .

[3]  Karl K. Sabelfeld.  Monte Carlo Methods in Boundary Value Problems. , 1991 .

[4]  Petros Drineas,et al.  An Experimental Evaluation of a Monte-Carlo Algorithm for Singular Value Decomposition , 2001, Panhellenic Conference on Informatics.

[5]  Santosh S. Vempala,et al.  The Random Projection Method , 2005, DIMACS Series in Discrete Mathematics and Theoretical Computer Science.

[6]  Karl K. Sabelfeld,et al.  Random Walk on Fixed Spheres for Laplace and Lamé equations , 2006, Monte Carlo Methods Appl..

[7]  V. Rokhlin,et al.  A fast randomized algorithm for the approximation of matrices ✩ , 2007 .

[8]  Aneta Karaivanova,et al.  Robustness and applicability of Markov chain Monte Carlo algorithms for eigenvalue problems , 2008 .

[9]  Christopher K. I. Williams,et al.  Unsupervised Learning of Multiple Aspects of Moving Objects from Video , 2005, Panhellenic Conference on Informatics.

[10]  Per-Gunnar Martinsson,et al.  Randomized algorithms for the low-rank approximation of matrices , 2007, Proceedings of the National Academy of Sciences.

[11]  Martin J. Mohlenkamp,et al.  Algorithms for Numerical Analysis in High Dimensions , 2005, SIAM J. Sci. Comput..

[12]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[13]  Ivan Tomov Dimov,et al.  Monte Carlo Numerical Treatment of Large Linear Algebra Problems , 2007, International Conference on Computational Science.

[14]  J. Hammersley,et al.  Monte Carlo Methods , 1965 .

[15]  PDEsK. K. Sabelfeld.  Expansion of random boundary ex itations for ellipti , 2007 .

[16]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[17]  C. Eckart,et al.  A principal axis transformation for non-hermitian matrices , 1939 .

[18]  Jack Dongarra,et al.  Computational Science - ICCS 2007, 7th International Conference, Beijing, China, May 27 - 30, 2007, Proceedings, Part III , 2007, ICCS.

[19]  G. W. Stewart,et al.  On the Early History of the Singular Value Decomposition , 1993, SIAM Rev..

[20]  Mads Nielsen,et al.  Computer Vision — ECCV 2002 , 2002, Lecture Notes in Computer Science.

[21]  M. Kobayashi,et al.  Estimation of singular values of very large matrices using random sampling , 2001 .

[22]  Petros Drineas,et al.  Pass efficient algorithms for approximating large matrices , 2003, SODA '03.

[23]  Alan M. Frieze,et al.  Fast monte-carlo algorithms for finding low-rank approximations , 2004, JACM.

[24]  Alan M. Frieze,et al.  Clustering Large Graphs via the Singular Value Decomposition , 2004, Machine Learning.

[25]  K. Phoon,et al.  Simulation of strongly non-Gaussian processes using Karhunen–Loeve expansion , 2005 .

[26]  V. Rokhlin,et al.  A randomized algorithm for the approximation of matrices , 2006 .

[27]  K. Sabelfeld,et al.  Random Walks on Boundary for Solving PDEs , 1994 .

[28]  Gene H. Golub,et al.  Matrix computations , 1983 .

[29]  Petros Drineas,et al.  Fast Monte-Carlo algorithms for approximate matrix multiplication , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[30]  Neil Muller,et al.  Singular Value Decomposition, Eigenfaces, and 3D Reconstructions , 2004, SIAM Rev..

[31]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[32]  Karl K. Sabelfeld,et al.  Sparsified Randomization Algorithms for large systems of linear equations and a new version of the Random Walk on Boundary method , 2009, Monte Carlo Methods Appl..

[33]  John S. Allen.  An Introductory Course , 1935 .

[34]  Karl K. Sabelfeld,et al.  Sparsified Randomization algorithms for low rank approximations and applications to integral equations and inhomogeneous random field simulation , 2011, Math. Comput. Simul..

[35]  Vera Pawlowsky-Glahn,et al.  Statistical Modeling , 2007, Encyclopedia of Social Network Analysis and Mining.

[36]  G. Strang.  The Fundamental Theorem of Linear Algebra , 1993 .

[37]  V. Rokhlin.  Rapid solution of integral equations of classical potential theory , 1985 .

[38]  Dimitris Achlioptas,et al.  Fast computation of low-rank matrix approximations , 2007, JACM.

[39]  Petros Drineas,et al.  Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication , 2006, SIAM J. Comput..

[40]  Erich Kaltofen,et al.  On randomized Lanczos algorithms , 1997, ISSAC.

[41]  Bernard Chazelle,et al.  The Fast Johnson--Lindenstrauss Transform and Approximate Nearest Neighbors , 2009, SIAM J. Comput..

[42]  A. J. Walker.  New fast method for generating discrete random numbers with arbitrary frequency distributions , 1974 .

[43]  Mark Tygert,et al.  A Randomized Algorithm for Principal Component Analysis , 2008, SIAM J. Matrix Anal. Appl..