Tensor neural network models for tensor singular value decompositions

Tensor decompositions have become increasingly prevalent in recent years. Traditionally, tensors are represented or decomposed as a sum of rank-one outer products using either the CANDECOMP/PARAFAC, the Tucker model, or some variations thereof. The motivation of these decompositions is to find an approximate representation for a given tensor. The main propose of this paper is to develop two neural network models for finding an approximation based on t-product for a given third-order tensor. Theoretical analysis shows that each of the neural network models ensures the convergence performance. The computer simulation results further substantiate that the models can find effectively the left and right singular tensor subspace.

[1]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[2]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[3]  Misha Elena Kilmer,et al.  Stable Tensor Neural Networks for Rapid Deep Learning , 2018, ArXiv.

[4]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[5]  Wei Liu,et al.  Tensor Robust Principal Component Analysis with a New Tensor Nuclear Norm , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Gerhard Zielke,et al.  Report on test matrices for generalized inverses , 1986, Computing.

[7]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[8]  K. Wright.  Differential equations for the analytic singular value decomposition of a matrix , 1992 .

[9]  Zheng Bao,et al.  A cross-associative neural network for SVD of non-squared data matrix in signal processing , 2001, IEEE Trans. Neural Networks.

[10]  Andrzej Cichocki,et al.  Neural networks for computing best rank-one approximations of tensors and its applications , 2017, Neurocomputing.

[11]  Yimin Wei,et al.  Two finite-time convergent Zhang neural network models for time-varying complex matrix Drazin inverse , 2017 .

[12]  Gene H. Golub,et al.  Symmetric Tensors and Symmetric Tensor Rank , 2008, SIAM J. Matrix Anal. Appl..

[13]  Ivan V. Oseledets,et al.  Wedderburn Rank Reduction and Krylov Subspace Method for Tensor Approximation. Part 1: Tucker Case , 2010, SIAM J. Sci. Comput..

[14]  Othmar Koch,et al.  Dynamical Low-Rank Approximation , 2007, SIAM J. Matrix Anal. Appl..

[15]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[16]  Demetri Terzopoulos,et al.  Multilinear subspace analysis of image ensembles , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[17]  Yimin Wei,et al.  Modified gradient dynamic approach to the tensor complementarity problem , 2018, Optim. Methods Softw..

[18]  Berkant Savas,et al.  A Newton-Grassmann Method for Computing the Best Multilinear Rank-(r1, r2, r3) Approximation of a Tensor , 2009, SIAM J. Matrix Anal. Appl..

[19]  A. Bunse-Gerstner,et al.  Numerical computation of an analytic singular value decomposition of a matrix valued function , 1991 .

[20]  Misha Elena Kilmer,et al.  Third-Order Tensors as Operators on Matrices: A Theoretical and Computational Framework with Applications in Imaging , 2013, SIAM J. Matrix Anal. Appl..

[21]  Sun-Yuan Kung,et al.  Cross-correlation neural network models , 1994, IEEE Trans. Signal Process..

[22]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[23]  Liqun Qi,et al.  Neurodynamical Optimization , 2004, J. Glob. Optim..

[24]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[25]  Jerzy Zabczyk,et al.  Mathematical control theory - an introduction , 1992, Systems & Control: Foundations & Applications.

[26]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[27]  Reinhold Schneider,et al.  Dynamical Approximation by Hierarchical Tucker and Tensor-Train Tensors , 2013, SIAM J. Matrix Anal. Appl..

[28]  C. L. Nikias,et al.  Signal processing with higher-order spectra , 1993, IEEE Signal Processing Magazine.

[29]  Othmar Koch,et al.  Dynamical Tensor Approximation , 2010, SIAM J. Matrix Anal. Appl..

[30]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[31]  Karen S. Braman.  Third-Order Tensors as Linear Operators on a Space of Matrices , 2010 .

[32]  M. Hirsch,et al.  Differential Equations, Dynamical Systems, and Linear Algebra , 1974 .

[33]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[34]  Yimin Wei,et al.  Randomized algorithms for the approximations of Tucker and the tensor train decompositions , 2018, Advances in Computational Mathematics.

[35]  Zemin Zhang,et al.  Exact Tensor Completion Using t-SVD , 2015, IEEE Transactions on Signal Processing.

[36]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[37]  J. P. Lasalle.  The stability of dynamical systems , 1976 .

[38]  Yimin Wei,et al.  Generalized tensor function via the tensor singular value decomposition based on the T-product , 2019, Linear Algebra and its Applications.

[39]  Timo Eirola,et al.  On Smooth Decompositions of Matrices , 1999, SIAM J. Matrix Anal. Appl..

[40]  Jean-Franois Cardoso.  High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[41]  Simone G. O. Fiori,et al.  Singular Value Decomposition Learning on Double Stiefel Manifold , 2003, Int. J. Neural Syst..

[42]  Masashi Sugiyama,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives , 2017, Found. Trends Mach. Learn..

[43]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[44]  M. Kilmer,et al.  Factorization strategies for third-order tensors , 2011 .

[45]  Joos Vandewalle,et al.  A Grassmann-Rayleigh Quotient Iteration for Dimensionality Reduction in ICA , 2004, ICA.

[46]  Edgar Sanchez-Sinencio,et al.  Nonlinear switched capacitor 'neural' networks for optimization problems , 1990 .

[47]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[48]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[49]  Pierre Comon,et al.  Tensor Decompositions, State of the Art and Applications , 2002 .

[50]  Yimin Wei,et al.  Neural networks based approach solving multi-linear systems with M-tensors , 2019, Neurocomputing.

[51]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[52]  Sabine Van Huffel,et al.  Differential-geometric Newton method for the best rank-(R1, R2, R3) approximation of tensors , 2008, Numerical Algorithms.

[53]  Berkant Savas,et al.  Krylov-Type Methods for Tensor Computations , 2010, 1005.0683.