Randomized interpolative decomposition of separated representations

We introduce an algorithm to compute tensor interpolative decomposition (dubbed CTD-ID) for the reduction of the separation rank of Canonical Tensor Decompositions (CTDs). Tensor ID selects, for a user-defined accuracy ?, a near optimal subset of terms of a CTD to represent the remaining terms via a linear combination of the selected terms. CTD-ID can be used as an alternative to or in combination with the Alternating Least Squares (ALS) algorithm. We present examples of its use within a convergent iteration to compute inverse operators in high dimensions. We also briefly discuss the spectral norm as a computational alternative to the Frobenius norm in estimating approximation errors of tensor ID.We reduce the problem of finding tensor IDs to that of constructing interpolative decompositions of certain matrices. These matrices are generated via randomized projection of the terms of the given tensor. We provide cost estimates and several examples of the new approach to the reduction of separation rank.

[1]  Martin J. Mohlenkamp,et al.  Numerical operator calculus in higher dimensions , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[2]  R. Bro.  PARAFAC. Tutorial and applications , 1997 .

[3]  Reinhold Schneider,et al.  Low rank tensor recovery via iterative hard thresholding , 2016, ArXiv.

[4]  Petros Drineas,et al.  Pass efficient algorithms for approximating large matrices , 2003, SODA '03.

[5]  Santosh S. Vempala,et al.  Latent semantic indexing: a probabilistic analysis , 1998, PODS '98.

[6]  J. Ballani,et al.  Black box approximation of tensors in hierarchical Tucker format , 2013 .

[7]  S. Goreinov,et al.  A Theory of Pseudoskeleton Approximations , 1997 .

[8]  Rasmus Bro,et al.  A comparison of algorithms for fitting the PARAFAC model , 2006, Comput. Stat. Data Anal..

[9]  Santosh S. Vempala,et al.  Tensor decomposition and approximation schemes for constraint satisfaction problems , 2005, STOC '05.

[10]  Ming Gu,et al.  Efficient Algorithms for Computing a Strong Rank-Revealing QR Factorization , 1996, SIAM J. Sci. Comput..

[11]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[12]  C. Chui,et al.  Article in Press Applied and Computational Harmonic Analysis a Randomized Algorithm for the Decomposition of Matrices , 2022 .

[13]  Martin J. Mohlenkamp,et al.  Preliminary results on approximating a wavefunction as an unconstrained sum of Slater determinants , 2007 .

[14]  Gregory Beylkin,et al.  On Generalized Gaussian Quadratures for Exponentials and Their Applications , 2002 .

[15]  V. Rokhlin,et al.  A randomized algorithm for the approximation of matrices , 2006 .

[16]  Severnyi Kavkaz.  Pseudo-Skeleton Approximations by Matrices of Maximal Volume , 2022 .

[17]  Trac D. Tran,et al.  Tensor sparsification via a bound on the spectral norm of random tensors , 2010, ArXiv.

[18]  N. Cutland,et al.  On homogeneous chaos , 1991, Mathematical Proceedings of the Cambridge Philosophical Society.

[19]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[20]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[21]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[22]  Petros Drineas,et al.  Tensor-CUR decompositions for tensor-based data , 2006, KDD '06.

[23]  Michael W. Mahoney,et al.  A randomized algorithm for a tensor-based generalization of the singular value decomposition , 2007 .

[24]  Bülent Yener,et al.  Unsupervised Multiway Data Analysis: A Literature Survey , 2009, IEEE Transactions on Knowledge and Data Engineering.

[25]  Robert J. Harrison,et al.  MULTIRESOLUTION REPRESENTATION OF OPERATORS WITH BOUNDARY CONDITIONS ON SIMPLE DOMAINS , 2012 .

[26]  Alan M. Frieze,et al.  Fast monte-carlo algorithms for finding low-rank approximations , 2004, JACM.

[27]  Dimitris Achlioptas,et al.  Database-friendly random projections: Johnson-Lindenstrauss with binary coins , 2003, J. Comput. Syst. Sci..

[28]  G. Schulz.  Iterative Berechung der reziproken Matrix , 1933 .

[29]  Gene H. Golub,et al.  Rank-One Approximation to High Order Tensors , 2001, SIAM J. Matrix Anal. Appl..

[30]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[31]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[32]  Per-Gunnar Martinsson,et al.  Randomized algorithms for the low-rank approximation of matrices , 2007, Proceedings of the National Academy of Sciences.

[33]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[34]  Eugene E. Tyrtyshnikov,et al.  Incomplete Cross Approximation in the Mosaic-Skeleton Method , 2000, Computing.

[35]  Bernard Chazelle,et al.  Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform , 2006, STOC '06.

[36]  Per-Gunnar Martinsson,et al.  On the Compression of Low Rank Matrices , 2005, SIAM J. Sci. Comput..

[37]  V. Prasolov.  Problems and theorems in linear algebra , 1994 .

[38]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[39]  E. Tyrtyshnikov.  Mosaic-Skeleton approximations , 1996 .

[40]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[41]  Lek-Heng Lim,et al.  Singular values and eigenvalues of tensors: a variational approach , 2005, 1st IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 2005..

[42]  David J. Biagioni,et al.  Numerical construction of Green's functions in high dimensional elliptic problems with variable coefficients and analysis of renewable energy data via sparse and separable approximations , 2012 .

[43]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[44]  G. Beylkin,et al.  On the representation of operators in bases of compactly supported wavelets , 1992 .

[45]  Phillip A. Regalia,et al.  On the Best Rank-1 Approximation of Higher-Order Supersymmetric Tensors , 2001, SIAM J. Matrix Anal. Appl..

[46]  Christopher J. Hillar,et al.  Most Tensor Problems Are NP-Hard , 2009, JACM.

[47]  Martin J. Mohlenkamp.  Musings on multilinear fitting , 2013 .

[48]  VandewalleJoos,et al.  On the Best Rank-1 and Rank-(R1,R2,. . .,RN) Approximation of Higher-Order Tensors , 2000 .

[49]  Charalampos E. Tsourakakis.  MACH: Fast Randomized Tensor Decompositions , 2009, SDM.

[50]  Wolfgang Dahmen,et al.  Wavelets in Numerical Analysis , 2005 .

[51]  James Demmel.  Accurate Singular Value Decompositions of Structured Matrices , 2000, SIAM J. Matrix Anal. Appl..

[52]  Dimitris Achlioptas,et al.  Fast computation of low rank matrix approximations , 2001, STOC '01.

[53]  Martin J. Mohlenkamp,et al.  Multivariate Regression and Machine Learning with Sums of Separable Functions , 2009, SIAM J. Sci. Comput..

[54]  Mark Rudelson,et al.  Sampling from large matrices: An approach through geometric functional analysis , 2005, JACM.

[55]  G. Beylkin,et al.  On approximation of functions by exponential sums , 2005 .

[56]  Tamás Sarlós,et al.  Improved Approximation Algorithms for Large Matrices via Random Projections , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[57]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[58]  V. Rokhlin,et al.  A fast randomized algorithm for the approximation of matrices ✩ , 2007 .

[59]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[60]  Martin J. Mohlenkamp,et al.  Algorithms for Numerical Analysis in High Dimensions , 2005, SIAM J. Sci. Comput..

[61]  R. Coifman,et al.  Fast wavelet transforms and numerical algorithms I , 1991 .

[62]  Tamara G. Kolda,et al.  Shifted Power Method for Computing Tensor Eigenpairs , 2010, SIAM J. Matrix Anal. Appl..

[63]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .