Very Large-Scale Singular Value Decomposition Using Tensor Train Networks

We propose new algorithms for singular value decomposition (SVD) of large-scale matrices based on a low-rank tensor approximation technique called the tensor train (TT) format. The proposed algorithms can compute a few extreme (i.e., largest or smallest) singular values and corresponding singular vectors for large-scale structured matrices given in a TT format. The computational complexity of the proposed methods scales logarithmically with the matrix size under the assumption that both the matrix and the singular vectors admit approximate low-rank TT decompositions. The proposed methods, which are called the alternating least squares SVD (ALS-SVD) and modified alternating least squares SVD (MALS-SVD), compute the left and right singular vectors approximately through block TT decompositions. A large-scale optimization problem is reduced to sequential small-scale optimization problems, and each core tensor of the block TT decompositions can be updated by applying any standard SVD method. The optimal ranks ...

[1]  Bart Vandereycken,et al.  The geometry of algorithms using hierarchical tensors , 2013, Linear Algebra and its Applications.

[2]  F. Verstraete,et al.  Renormalization algorithms for Quantum-Many Body Systems in two and higher dimensions , 2004, cond-mat/0407066.

[3]  Y. Saad.  Numerical Methods for Large Eigenvalue Problems , 2011 .

[4]  S. V. Dolgov,et al.  ALTERNATING MINIMAL ENERGY METHODS FOR LINEAR SYSTEMS IN HIGHER DIMENSIONS∗ , 2014 .

[5]  Andrew V. Knyazev,et al.  Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method , 2001, SIAM J. Sci. Comput..

[6]  Yinchu Zhu,et al.  Breaking the curse of dimensionality in regression , 2017, ArXiv.

[7]  D. Sorensen.  Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[8]  W. Hackbusch,et al.  A New Scheme for the Tensor Representation , 2009 .

[9]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[10]  J. Ballani,et al.  Black box approximation of tensors in hierarchical Tucker format , 2013 .

[11]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[12]  Martin J. Mohlenkamp,et al.  Numerical operator calculus in higher dimensions , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[13]  Yin Zhang,et al.  Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions , 2013, SIAM J. Sci. Comput..

[14]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[15]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[16]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[17]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[18]  Boris N. Khoromskij,et al.  Computation of extreme eigenvalues in higher dimensions using block tensor train format , 2013, Comput. Phys. Commun..

[19]  Andrzej Cichocki,et al.  Fundamental Tensor Operations for Large-Scale Data Analysis in Tensor Train Formats , 2014, ArXiv.

[20]  D. Sorensen,et al.  4. The Implicitly Restarted Arnoldi Method , 1998 .

[21]  Alan M. Frieze,et al.  Fast Monte-Carlo algorithms for finding low-rank approximations , 1998, Proceedings 39th Annual Symposium on Foundations of Computer Science (Cat. No.98CB36280).

[22]  Jennifer Seberry,et al.  The Strong Kronecker Product , 1994, J. Comb. Theory, Ser. A.

[23]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .

[24]  VLADIMIR A. KAZEEV,et al.  Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse , 2012, SIAM J. Matrix Anal. Appl..

[25]  B. Khoromskij,et al.  DMRG+QTT approach to computation of the ground state for the molecular Schrödinger operator , 2010 .

[26]  Reinhold Schneider,et al.  The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format , 2012, SIAM J. Sci. Comput..

[27]  G. Vidal.  Efficient classical simulation of slightly entangled quantum computations. , 2003, Physical review letters.

[28]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[29]  K. Fan.  On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations: II. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[30]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[31]  Yang Qi,et al.  On the geometry of tensor network states , 2011, Quantum Inf. Comput..

[32]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[33]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[34]  Alan M. Frieze,et al.  Fast monte-carlo algorithms for finding low-rank approximations , 2004, JACM.

[35]  James Demmel,et al.  Applied Numerical Linear Algebra , 1997 .

[36]  Daniel Kressner,et al.  Low-Rank Tensor Methods with Subspace Correction for Symmetric Eigenvalue Problems , 2014, SIAM J. Sci. Comput..

[37]  Boris N. Khoromskij,et al.  Two-Level QTT-Tucker Format for Optimized Tensor Calculus , 2013, SIAM J. Matrix Anal. Appl..

[38]  О. С. Лебедева.  Tensor conjugate-gradient-type method for Rayleigh quotient minimization in block QTT format , 2011 .

[39]  G. Golub,et al.  Tracking a few extreme singular values and vectors in signal processing , 1990, Proc. IEEE.

[40]  Luis Mateus Rocha,et al.  Singular value decomposition and principal component analysis , 2003 .

[41]  U. Schollwoeck.  The density-matrix renormalization group in the age of matrix product states , 2010, 1008.3477.

[42]  Reinhold Schneider,et al.  Optimization problems in contracted tensor networks , 2011, Comput. Vis. Sci..

[43]  Thomas Huckle,et al.  Subspace Iteration Methods in terms of Matrix Product States , 2012 .

[44]  Thomas Mach,et al.  Computing Inner Eigenvalues of Matrices in Tensor Train Matrix Format , 2013 .

[45]  White,et al.  Density-matrix algorithms for quantum renormalization groups. , 1993, Physical review. B, Condensed matter.

[46]  Vladimir A. Kazeev,et al.  Multilevel Toeplitz Matrices Generated by Tensor-Structured Vectors and Convolution with Logarithmic Complexity , 2013, SIAM J. Sci. Comput..

[47]  Reinhold Schneider,et al.  Dynamical Approximation by Hierarchical Tucker and Tensor-Train Tensors , 2013, SIAM J. Matrix Anal. Appl..

[48]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..