Computing low-rank approximations of large-scale matrices with the Tensor Network randomized SVD

We propose a new algorithm for the computation of a singular value decomposition (SVD) low-rank approximation of a matrix in the Matrix Product Operator (MPO) format, also called the Tensor Train Matrix format. Our tensor network randomized SVD (TNrSVD) algorithm is an MPO implementation of the randomized SVD algorithm that is able to compute dominant singular values and their corresponding singular vectors. In contrast to the state-of-the-art tensor-based alternating least squares SVD (ALS-SVD) and modified alternating least squares SVD (MALS-SVD) matrix approximation methods, TNrSVD can be up to 17 times faster while achieving the same accuracy. In addition, our TNrSVD algorithm also produces accurate approximations in particular cases where both ALS-SVD and MALS-SVD fail to converge. We also propose a new algorithm for the fast conversion of a sparse matrix into its corresponding MPO form, which is up to 509 times faster than the standard Tensor Train SVD (TT-SVD) method while achieving machine precision accuracy. The efficiency and accuracy of both algorithms are demonstrated in numerical experiments.

[1]  Gene H. Golub,et al.  Singular value decomposition and least squares solutions , 1970, Milestones in Matrix Computation.

[2]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[3]  I. McCulloch,et al.  Generic construction of efficient matrix product operators , 2016, 1611.02498.

[4]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[5]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[6]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[7]  Ivan V. Oseledets,et al.  Approximation of 2d˟2d Matrices Using Tensor Decomposition , 2010, SIAM J. Matrix Anal. Appl..

[8]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[9]  Johan A. K. Suykens,et al.  Parallelized Tensor Train Learning of Polynomial Classifiers , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[10]  U. Schollwoeck.  The density-matrix renormalization group in the age of matrix product states , 2010, 1008.3477.

[11]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[12]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .

[13]  Timothy A. Davis,et al.  The university of Florida sparse matrix collection , 2011, TOMS.

[14]  Ngai Wong,et al.  Tensor Network alternating linear scheme for MIMO Volterra system identification , 2016, Autom..

[15]  Yaohang Li,et al.  Single-Pass PCA of Large High-Dimensional Data , 2017, IJCAI.

[16]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[17]  Ivan V. Oseledets,et al.  Fast adaptive interpolation of multi-dimensional arrays in tensor train format , 2011, The 2011 International Workshop on Multidimensional (nD) Systems.

[18]  Andrzej Cichocki,et al.  Regularized Computation of Approximate Pseudoinverse of Large Matrices Using Low-Rank Tensor Train Decompositions , 2015, SIAM J. Matrix Anal. Appl..

[19]  Daniela Calvetti,et al.  Matrix methods in data mining and pattern recognition , 2009, Math. Comput..

[20]  Masashi Sugiyama,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives , 2017, Found. Trends Mach. Learn..

[21]  Andrzej Cichocki,et al.  Very Large-Scale Singular Value Decomposition Using Tensor Train Networks , 2014, ArXiv.

[22]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[23]  Ivan V. Oseledets,et al.  DMRG Approach to Fast Linear Algebra in the TT-Format , 2011, Comput. Methods Appl. Math..

[24]  Yaohang Li,et al.  Efficient randomized algorithms for adaptive low-rank factorizations of large matrices , 2016, ArXiv.

[25]  E. Cuthill,et al.  Reducing the bandwidth of sparse symmetric matrices , 1969, ACM '69.

[26]  Gene H. Golub,et al.  Calculating the singular values and pseudo-inverse of a matrix , 2007, Milestones in Matrix Computation.