Effective Tensor Sketching via Sparsification

In this article, we investigate effective sketching schemes via sparsification for high dimensional multilinear arrays or tensors. More specifically, we propose a novel tensor sparsification algorithm that retains a subset of the entries of a tensor in a judicious way, and prove that it can attain a given level of approximation accuracy in terms of tensor spectral norm with a much smaller sample complexity when compared with existing approaches. In particular, we show that for a <italic>k</italic>th order <inline-formula> <tex-math notation="LaTeX">$ {d}\times \cdots \times {d}$ </tex-math></inline-formula> cubic tensor of <italic>stable rank</italic> <inline-formula> <tex-math notation="LaTeX">$ {r}_{ {s}}$ </tex-math></inline-formula>, the sample size requirement for achieving a relative error <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> is, up to a logarithmic factor, of the order <inline-formula> <tex-math notation="LaTeX">$ {r}_{ {s}}^{1/2} {d}^{ {k}/2} /\varepsilon $ </tex-math></inline-formula> when <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> is relatively large, and <inline-formula> <tex-math notation="LaTeX">$ {r}_{ {s}} {d} /\varepsilon ^{2}$ </tex-math></inline-formula> and essentially optimal when <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> is sufficiently small. It is especially noteworthy that the sample size requirement for achieving a high accuracy is of an order independent of <italic>k</italic>. To further demonstrate the utility of our techniques, we also study how higher order singular value decomposition (HOSVD) of large tensors can be efficiently approximated via sparsification.

[1]  Charles Elkan,et al.  Fast Algorithms for Approximating the Singular Value Decomposition , 2011, TKDD.

[2]  Edo Liberty,et al.  Near-Optimal Entrywise Sampling for Data Matrices , 2013, NIPS.

[3]  M. Kobayashi,et al.  Estimation of singular values of very large matrices using random sampling , 2001 .

[4]  Mark W. Woolrich,et al.  Advances in functional and structural MR image analysis and implementation as FSL , 2004, NeuroImage.

[5]  W. Kahan,et al.  The Rotation of Eigenvectors by a Perturbation. III , 1970 .

[6]  Trac D. Tran,et al.  Tensor sparsification via a bound on the spectral norm of random tensors , 2010, ArXiv.

[7]  M. Newman,et al.  Finding community structure in very large networks. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[8]  Tom A. B. Snijders,et al.  Social Network Analysis , 2011, International Encyclopedia of Statistical Science.

[9]  Baoxin Li,et al.  Tensor completion for on-board compression of hyperspectral images , 2010, 2010 IEEE International Conference on Image Processing.

[10]  H. Lehrach,et al.  A Human Protein-Protein Interaction Network: A Resource for Annotating the Proteome , 2005, Cell.

[11]  Dimitris Achlioptas,et al.  Fast computation of low rank matrix approximations , 2001, STOC '01.

[12]  Petros Drineas,et al.  A note on element-wise matrix sparsification via a matrix-valued Bernstein inequality , 2010, Inf. Process. Lett..

[13]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[14]  Nikos D. Sidiropoulos,et al.  Tensor Decomposition for Signal Processing and Machine Learning , 2016, IEEE Transactions on Signal Processing.

[15]  David P. Woodruff.  Sketching as a Tool for Numerical Linear Algebra , 2014, Found. Trends Theor. Comput. Sci..

[16]  Sujay Sanghavi,et al.  A New Sampling Technique for Tensors , 2015, ArXiv.

[17]  S. Muthukrishnan,et al.  Subspace Sampling and Relative-Error Matrix Approximation: Column-Based Methods , 2006, APPROX-RANDOM.

[18]  Ming Yuan,et al.  On Polynomial Time Methods for Exact Low-Rank Tensor Completion , 2017, Found. Comput. Math..

[19]  Roman Orus,et al.  A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States , 2013, 1306.2164.

[20]  Ming Yuan,et al.  Incoherent Tensor Norms and Their Applications in Higher Order Tensor Completion , 2016, IEEE Transactions on Information Theory.

[21]  Akshay Krishnamurthy,et al.  Low-Rank Matrix and Tensor Completion via Adaptive Sampling , 2013, NIPS.

[22]  Jieping Ye,et al.  Tensor Completion for Estimating Missing Values in Visual Data , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[24]  Sanjeev Arora,et al.  A Fast Random Sampling Algorithm for Sparsifying Matrices , 2006, APPROX-RANDOM.

[25]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[26]  Ming Yuan,et al.  On Tensor Completion via Nuclear Norm Minimization , 2014, Foundations of Computational Mathematics.

[27]  Joseph T. Chang,et al.  Spectral biclustering of microarray data: coclustering genes and conditions. , 2003, Genome research.

[28]  Alan M. Frieze,et al.  Fast monte-carlo algorithms for finding low-rank approximations , 2004, JACM.

[29]  Chandler Davis.  The rotation of eigenvectors by a perturbation , 1963 .

[30]  Michael W. Berry,et al.  Large-Scale Sparse Singular Value Computations , 1992 .

[31]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..