A Rank Revealing Randomized Singular Value Decomposition (R3SVD) Algorithm for Low-rank Matrix Approximations

In this paper, we present a Rank Revealing Randomized Singular Value Decomposition (R3SVD) algorithm to incrementally construct a low-rank approximation of a potentially large matrix while adaptively estimating the appropriate rank that can capture most of the actions of the matrix. Starting from a low-rank approximation with an initial guessed rank, R3SVD adopts an orthogonal Gaussian sampling approach to obtain the dominant subspace within the leftover space, which is used to add up to the existing low-rank approximation. Orthogonal Gaussian sampling is repeated until an appropriate low-rank approximation with satisfactory accuracy, measured by the overall energy percentage of the original matrix, is obtained. While being a fast algorithm, R3SVD is also a memory-aware algorithm where the computational process can be decomposed into a series of sampling tasks that use constant amount of memory. Numerical examples in image compression and matrix completion are used to demonstrate the effectiveness of R3SVD in low-rank approximation.

[1]  I. Jolliffe.  Principal Component Analysis , 2002 .

[2]  Yaohang Li,et al.  GPU Accelerated Randomized Singular Value Decomposition and Its Application in Image Compression , 2015 .

[3]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[4]  Yaohang Li,et al.  March Madness Prediction : A Matrix Completion Approach , 2015 .

[5]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[6]  Per-Gunnar Martinsson,et al.  Randomized algorithms for the low-rank approximation of matrices , 2007, Proceedings of the National Academy of Sciences.

[7]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[8]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[9]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[10]  Xuelong Li,et al.  Matrix completion by Truncated Nuclear Norm Regularization , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Petros Drineas,et al.  An Experimental Evaluation of a Monte-Carlo Algorithm for Singular Value Decomposition , 2001, Panhellenic Conference on Informatics.

[12]  Per-Gunnar Martinsson,et al.  RSVDPACK: Subroutines for computing partial singular value decompositions via randomized sampling on single core, multi core, and GPU architectures , 2015, ArXiv.

[13]  Yaohang Li,et al.  Monte Carlo Methods and Their Applications in Big Data Analysis , 2015 .

[14]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[15]  Petros Drineas,et al.  FAST MONTE CARLO ALGORITHMS FOR MATRICES II: COMPUTING A LOW-RANK APPROXIMATION TO A MATRIX∗ , 2004 .

[16]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[17]  Petros Drineas,et al.  On the Nyström Method for Approximating a Gram Matrix for Improved Kernel-Based Learning , 2005, J. Mach. Learn. Res..

[18]  Arak M. Mathai,et al.  Characterizations of the normal probability law , 1977 .

[19]  Zuowei Shen,et al.  Robust video denoising using low rank matrix completion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Nathan Halko,et al.  Randomized methods for computing low-rank approximations of matrices , 2012 .

[21]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[22]  Ilse C. F. Ipsen,et al.  Importance Sampling for a Monte Carlo Matrix Multiplication Algorithm, with Application to Information Retrieval , 2011, SIAM J. Sci. Comput..

[23]  Charles E. Heckler,et al.  Applied Multivariate Statistical Analysis , 2005, Technometrics.

[24]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[25]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[26]  Michael W. Mahoney.  Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[27]  R. Larsen.  Lanczos Bidiagonalization With Partial Reorthogonalization , 1998 .

[28]  Per-Gunnar Martinsson,et al.  A Randomized Blocked Algorithm for Efficiently Computing Rank-revealing Factorizations of Matrices , 2015, SIAM J. Sci. Comput..

[29]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[30]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[31]  Benjamin Recht,et al.  A Simpler Approach to Matrix Completion , 2009, J. Mach. Learn. Res..