Neural networks for computing best rank-one approximations of tensors and its applications

Abstract This paper presents the neural dynamical network to compute a best rank-one approximation of a real-valued tensor. We implement the neural network model by the ordinary differential equations (ODE), which is a class of continuous-time recurrent neural network. Several new properties of solutions for the neural network are established. We prove that the locally asymptotic stability of solutions for ODE by constructive an appropriate Lyapunov function under mild conditions. Furthermore, we also discuss how to use the proposed neural networks for solving the tensor eigenvalue problem including the tensor H-eigenvalue problem, the tensor Z-eigenvalue problem, and the generalized eigenvalue problem with symmetric-definite tensor pairs. Finally, we generalize the proposed neural networks to the computation of the restricted singular values and the associated restricted singular vectors of real-valued tensors. We illustrate and validate theoretical results via numerical simulations.

[1]  Liqun Qi,et al.  The Best Rank-One Approximation Ratio of a Tensor Space , 2011, SIAM J. Matrix Anal. Appl..

[2]  Jiawang Nie,et al.  All Real Eigenvalues of Symmetric Tensors , 2014, SIAM J. Matrix Anal. Appl..

[3]  Tamara G. Kolda,et al.  Shifted Power Method for Computing Tensor Eigenpairs , 2010, SIAM J. Matrix Anal. Appl..

[4]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[5]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[6]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[7]  Edgar Sanchez-Sinencio,et al.  Nonlinear switched capacitor 'neural' networks for optimization problems , 1990 .

[8]  Gene H. Golub,et al.  Symmetric Tensors and Symmetric Tensor Rank , 2008, SIAM J. Matrix Anal. Appl..

[9]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[10]  Nikola Samardzija,et al.  A neural network for computing eigenvectors and eigenvalues , 1991, Biological Cybernetics.

[11]  S. Friedland.  Best rank one approximation of real symmetric tensors can be chosen symmetric , 2011, 1110.5689.

[12]  Lek-Heng Lim,et al.  Singular values and eigenvalues of tensors: a variational approach , 2005, 1st IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 2005..

[13]  Yong Luo,et al.  Tensor Canonical Correlation Analysis for Multi-View Dimension Reduction , 2015, IEEE Trans. Knowl. Data Eng..

[14]  Liqun Qi,et al.  On the successive supersymmetric rank‐1 decomposition of higher‐order supersymmetric tensors , 2007, Numer. Linear Algebra Appl..

[15]  Liqun Qi,et al.  Geometric Measure of Entanglement and U-Eigenvalues of Tensors , 2014, SIAM J. Matrix Anal. Appl..

[16]  Gene H. Golub,et al.  Rank-One Approximation to High Order Tensors , 2001, SIAM J. Matrix Anal. Appl..

[17]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[18]  Zheng-Hai Huang,et al.  Finding the extreme Z‐eigenvalues of tensors via a sequential semidefinite programming method , 2013, Numer. Linear Algebra Appl..

[19]  Berkant Savas,et al.  A Newton-Grassmann Method for Computing the Best Multilinear Rank-(r1, r2, r3) Approximation of a Tensor , 2009, SIAM J. Matrix Anal. Appl..

[20]  Bruno Sudret,et al.  Uncertainty quantification in high-dimensional spaces with low-rank tensor approximations , 2015 .

[21]  Liqun Qi,et al.  D-eigenvalues of diffusion kurtosis tensors , 2008 .

[22]  Bo Yu,et al.  Orthogonal Low Rank Tensor Approximation: Alternating Least Squares Method and Its Global Convergence , 2015, SIAM J. Matrix Anal. Appl..

[23]  Kung-Ching Chang,et al.  Some variational principles for Z-eigenvalues of nonnegative tensors , 2013 .

[24]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[25]  Liqun Qi,et al.  Eigenvalues of a real supersymmetric tensor , 2005, J. Symb. Comput..

[26]  David J. Schwab,et al.  Supervised Learning with Quantum-Inspired Tensor Networks , 2016, ArXiv.

[27]  Gene H. Golub,et al.  Matrix computations , 1983 .

[28]  Shmuel Friedland,et al.  The Number of Singular Vector Tuples and Uniqueness of Best Rank-One Approximation of Tensors , 2012, Found. Comput. Math..

[29]  Jean-Franois Cardoso.  High-Order Contrasts for Independent Component Analysis , 1999, Neural Computation.

[30]  Bart De Moor,et al.  On the Computation of the Restricted Singular Value Decomposition via the Cosine-Sine Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[31]  Liping Chen,et al.  Computing Tensor Eigenvalues via Homotopy Methods , 2015, SIAM J. Matrix Anal. Appl..

[32]  YUNING YANG,et al.  Rank-1 Tensor Properties with Applications to a Class of Tensor Optimization Problems , 2016, SIAM J. Optim..

[33]  Pierre Comon,et al.  Tensor Decompositions, State of the Art and Applications , 2002 .

[34]  Simone G. O. Fiori,et al.  Singular Value Decomposition Learning on Double Stiefel Manifold , 2003, Int. J. Neural Syst..

[35]  Yiju Wang,et al.  On the best rank-1 approximation to higher-order symmetric tensors , 2007, Math. Comput. Model..

[36]  Liqun Qi,et al.  Neurodynamical Optimization , 2004, J. Glob. Optim..

[37]  Pedro J. Zufiria,et al.  Generalized neural networks for spectral analysis: dynamics and Liapunov functions , 2004, Neural Networks.

[38]  Tamara G. Kolda,et al.  An Adaptive Shifted Power Method for Computing Generalized Tensor Eigenpairs , 2014, SIAM J. Matrix Anal. Appl..

[39]  Zheng Bao,et al.  A cross-associative neural network for SVD of non-squared data matrix in signal processing , 2001, IEEE Trans. Neural Networks.

[40]  Kung-Ching Chang,et al.  Perron-Frobenius theorem for nonnegative tensors , 2008 .

[41]  Chen Ling,et al.  The Best Rank-1 Approximation of a Symmetric Tensor and Related Spherical Optimization Problems , 2012, SIAM J. Matrix Anal. Appl..

[42]  Sun-Yuan Kung,et al.  Cross-correlation neural network models , 1994, IEEE Trans. Signal Process..

[43]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[44]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[45]  Phillip A. Regalia,et al.  On the Best Rank-1 Approximation of Higher-Order Supersymmetric Tensors , 2001, SIAM J. Matrix Anal. Appl..

[46]  Yimin Wei,et al.  Generalized Tensor Eigenvalue Problems , 2015, SIAM J. Matrix Anal. Appl..

[47]  Yao-Lin Jiang,et al.  On the Uniqueness and Perturbation to the Best Rank-One Approximation of a Tensor , 2015, SIAM J. Matrix Anal. Appl..

[48]  Fei Wang,et al.  Z-eigenvalue methods for a global polynomial optimization problem , 2009, Math. Program..

[49]  Kung-Ching Chang,et al.  On eigenvalue problems of real symmetric tensors , 2009 .

[50]  Li Wang,et al.  Semidefinite Relaxations for Best Rank-1 Tensor Approximations , 2013, SIAM J. Matrix Anal. Appl..

[51]  VandewalleJoos,et al.  On the Best Rank-1 and Rank-(R1,R2,. . .,RN) Approximation of Higher-Order Tensors , 2000 .

[52]  C. L. Nikias,et al.  Signal processing with higher-order spectra , 1993, IEEE Signal Processing Magazine.

[53]  Wei Wu,et al.  Dynamical System for Computing Largest Generalized Eigenvalue , 2006, ISNN.

[54]  Yimin Wei,et al.  Perturbation bounds of tensor eigenvalue and singular value problems with even order , 2016 .

[55]  C. Loan.  Generalizing the Singular Value Decomposition , 1976 .

[56]  Andrzej Cichocki,et al.  Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions , 2016, Found. Trends Mach. Learn..

[57]  Andrzej Cichocki,et al.  Neural network for singular value decomposition , 1992 .

[58]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[59]  Charles Van Loan,et al.  Block tensors and symmetric embeddings , 2010, ArXiv.

[60]  Jianping Li,et al.  Notes on "Recurrent neural network model for computing largest and smallest generalized eigenvalue" , 2010, Neurocomputing.