A Fast Implementation of Singular Value Thresholding Algorithm using Recycling Rank Revealing Randomized Singular Value Decomposition

In this paper, we present a fast implementation of the Singular Value Thresholding (SVT) algorithm for matrix completion. A rank-revealing randomized singular value decomposition (R3SVD) algorithm is used to adaptively carry out partial singular value decomposition (SVD) to fast approximate the SVT operator given a desired, fixed precision. We extend the R3SVD algorithm to a recycling rank revealing randomized singular value decomposition (R4SVD) algorithm by reusing the left singular vectors obtained from the previous iteration as the approximate basis in the current iteration, where the computational cost for partial SVD at each SVT iteration is significantly reduced. A simulated annealing style cooling mechanism is employed to adaptively adjust the low-rank approximation precision threshold as SVT progresses. Our fast SVT implementation is effective in both large and small matrices, which is demonstrated in matrix completion applications including image recovery and movie recommendation system.

[1]  Yong-Jin Liu,et al.  An implementable proximal point algorithmic framework for nuclear norm minimization , 2012, Math. Program..

[2]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[3]  Yaohang Li,et al.  Efficient randomized algorithms for adaptive low-rank factorizations of large matrices , 2016, ArXiv.

[4]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[5]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[6]  Jian Dong,et al.  Accelerated low-rank visual recovery by random projection , 2011, CVPR 2011.

[7]  F. Maxwell Harper,et al.  The MovieLens Datasets: History and Context , 2016, TIIS.

[8]  Yaohang Li,et al.  Hybrid parallel tempering and simulated annealing method , 2009, Appl. Math. Comput..

[9]  Charles Elkan,et al.  Fast Algorithms for Approximating the Singular Value Decomposition , 2011, TKDD.

[10]  S. Yun,et al.  An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems , 2009 .

[11]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[12]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[13]  Shiqian Ma,et al.  Fixed point and Bregman iterative methods for matrix rank minimization , 2009, Math. Program..

[14]  G. Sapiro,et al.  A collaborative framework for 3D alignment and classification of heterogeneous subvolumes in cryo-electron tomography. , 2013, Journal of structural biology.

[15]  Emmanuel J. Candès,et al.  Simple bounds for recovering low-complexity models , 2011, Math. Program..

[16]  Yi Ma,et al.  The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices , 2010, Journal of structural biology.

[17]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[18]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[19]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[20]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[21]  Yasuyuki Matsushita,et al.  Fast randomized Singular Value Thresholding for Nuclear Norm Minimization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  S. Osher,et al.  Fast Singular Value Thresholding without Singular Value Decomposition , 2013 .

[23]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).