On algorithms for and computing with the tensor ring decomposition

Tensor decompositions such as the canonical format and the tensor train format have been widely utilized to reduce storage costs and operational complexities for high-dimensional data, achieving linear scaling with the input dimension instead of exponential scaling. In this paper, we investigate even lower storage-cost representations in the tensor ring format, which is an extension of the tensor train format with variable end-ranks. Firstly, we introduce two algorithms for converting a tensor in full format to tensor ring format with low storage cost. Secondly, we detail a rounding operation for tensor rings and show how this requires new definitions of common linear algebra operations in the format to obtain storage-cost savings. Lastly, we introduce algorithms for transforming the graph structure of graph-based tensor formats, with orders of magnitude lower complexity than existing literature. The efficiency of all algorithms is demonstrated on a number of numerical examples, and in certain cases, we demonstrate significantly higher compression ratios when compared to previous approaches to using the tensor ring format.

[1]  Frank Verstraete,et al.  Matrix product state representations , 2006, Quantum Inf. Comput..

[2]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[3]  Lek-Heng Lim,et al.  Tensor network ranks , 2018, 1801.02662.

[4]  Yingzhou Li,et al.  Tensor Ring Decomposition: Energy Landscape and One-loop Convergence of Alternating Least Squares , 2019, 1905.07101.

[5]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[6]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[7]  V. Aggarwal,et al.  Efficient Low Rank Tensor Ring Completion , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  S. V. DOLGOV,et al.  Fast Solution of Parabolic Problems in the Tensor Train/Quantized Tensor Train Format with Initial Application to the Fokker-Planck Equation , 2012, SIAM J. Sci. Comput..

[9]  Boris N. Khoromskij,et al.  Fast tensor method for summation of long-range potentials on 3D lattices with defects , 2016, Numer. Linear Algebra Appl..

[10]  Yifan Sun,et al.  Wide Compression: Tensor Ring Nets , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Yang Qi,et al.  On the geometry of tensor network states , 2011, Quantum Inf. Comput..

[12]  Kim Batselier.  The trouble with tensor ring decompositions , 2018, ArXiv.

[13]  F. L. Hitchcock.  The Expression of a Tensor or a Polyadic as a Sum of Products , 1927 .

[14]  Jianting Cao,et al.  High-order tensor completion via gradient-based optimization under tensor train format , 2019, Signal Process. Image Commun..

[15]  Sameer A. Nene,et al.  Columbia Object Image Library (COIL100) , 1996 .

[16]  Chao Li,et al.  Rank Minimization on Tensor Ring: A New Paradigm in Scalable Tensor Decomposition and Completion , 2018, ArXiv.

[17]  Boris N. Khoromskij,et al.  Use of tensor formats in elliptic eigenvalue problems , 2012, Numer. Linear Algebra Appl..

[18]  Lexing Ying,et al.  Efficient Construction of Tensor Ring Representations from Sampling , 2017, Multiscale Model. Simul..

[19]  Roman Orus,et al.  A Practical Introduction to Tensor Networks: Matrix Product States and Projected Entangled Pair States , 2013, 1306.2164.

[20]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[21]  Boris N. Khoromskij,et al.  Simultaneous state-time approximation of the chemical master equation using tensor product formats , 2015, Numer. Linear Algebra Appl..

[22]  Kishore Kumar Naraparaju,et al.  A note on tensor chain approximation , 2012, Comput. Vis. Sci..

[23]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[24]  Lars Grasedyck,et al.  Distributed hierarchical SVD in the Hierarchical Tucker format , 2018, Numer. Linear Algebra Appl..

[25]  Stefan Handschuh.  Changing the topology of Tensor Networks , 2012 .

[26]  Eugene E. Tyrtyshnikov,et al.  Cross approximation in tensor electron density computations , 2010, Numer. Linear Algebra Appl..

[27]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[28]  Nico Vervliet,et al.  Linear systems with a canonical polyadic decomposition constrained solution: Algorithms and applications , 2018, Numer. Linear Algebra Appl..

[29]  B. Khoromskij.  Tensors-structured Numerical Methods in Scientific Computing: Survey on Recent Advances , 2012 .

[30]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[31]  Masashi Sugiyama,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives , 2017, Found. Trends Mach. Learn..

[32]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[33]  Masashi Sugiyama,et al.  Learning Efficient Tensor Representations with Ring-structured Networks , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[34]  Liqing Zhang,et al.  Tensor Ring Decomposition , 2016, ArXiv.

[35]  E. Lieb,et al.  Valence bond ground states in isotropic quantum antiferromagnets , 1988 .

[36]  Jianfeng Lu,et al.  Tensor Ring Decomposition: Optimization Landscape and One-loop Convergence of Alternating Least Squares , 2019, SIAM J. Matrix Anal. Appl..

[37]  U. Schollwoeck.  The density-matrix renormalization group in the age of matrix product states , 2010, 1008.3477.

[38]  B. Khoromskij.  O(dlog N)-Quantics Approximation of N-d Tensors in High-Dimensional Numerical Modeling , 2011 .