On Fast Computation of Gradients for CANDECOMP/PARAFAC Algorithms

Product between mode-$n$ unfolding $\bY_{(n)}$ of an $N$-D tensor $\tY$ and Khatri-Rao products of $(N-1)$ factor matrices $\bA^{(m)}$, $m = 1,..., n-1, n+1, ..., N$ exists in algorithms for CANDECOMP/PARAFAC (CP). If $\tY$ is an error tensor of a tensor approximation, this product is the gradient of a cost function with respect to factors, and has the largest workload in most CP algorithms. In this paper, a fast method to compute this product is proposed. Experimental verification shows that the fast CP gradient can accelerate the CP_ALS algorithm 2 times and 8 times faster for factorizations of 3-D and 4-D tensors, and the speed-up ratios can be 20-30 times for higher dimensional tensors.

[1]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[2]  Rasmus Bro,et al.  Multilinear Models, Iterative Methods , 2020, Comprehensive Chemometrics.

[3]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[4]  Lieven De Lathauwer,et al.  Tensor-based techniques for the blind separation of DS-CDMA signals , 2007, Signal Process..

[5]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[6]  Henk A. L. Kiers,et al.  A three–step algorithm for CANDECOMP/PARAFAC analysis of large data sets with multicollinearity , 1998 .

[7]  Rasmus Bro,et al.  The N-way Toolbox for MATLAB , 2000 .

[8]  P. Paatero.  A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis , 1997 .

[9]  Laurent Albera,et al.  Iterative methods for the canonical decomposition of multi-way arrays: Application to blind underdetermined mixture identification , 2011, Signal Process..

[10]  Tamara G. Kolda,et al.  Efficient MATLAB Computations with Sparse and Factored Tensors , 2007, SIAM J. Sci. Comput..

[11]  Lieven De Lathauwer,et al.  Blind Identification of Underdetermined Mixtures by Simultaneous Matrix Diagonalization , 2008, IEEE Transactions on Signal Processing.

[12]  Daniel M. Dunlavy,et al.  A scalable optimization approach for fitting canonical tensor decompositions , 2011 .

[13]  Lieven De Lathauwer,et al.  A Link between the Canonical Decomposition in Multilinear Algebra and Simultaneous Matrix Diagonalization , 2006, SIAM J. Matrix Anal. Appl..

[14]  Laurent Albera,et al.  Semi-algebraic canonical decomposition of multi-way arrays and Joint Eigenvalue Decomposition , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[16]  Lars Kai Hansen,et al.  Parallel Factor Analysis as an exploratory tool for wavelet transformed event-related EEG , 2006, NeuroImage.

[17]  Pierre Comon,et al.  Enhanced Line Search: A Novel Method to Accelerate PARAFAC , 2008, SIAM J. Matrix Anal. Appl..

[18]  Michael W. Berry,et al.  Discussion Tracking in Enron Email using PARAFAC. , 2008 .

[19]  F. L. Hitchcock.  Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[20]  Tamara G. Kolda,et al.  Categories and Subject Descriptors: G.4 [Mathematics of Computing]: Mathematical Software— , 2022 .

[21]  Boris N. Khoromskij,et al.  Mathematik in den Naturwissenschaften Leipzig Tensor-Product Approximation to Operators and Functions in High Dimensions , 2007 .

[22]  M. Haardt,et al.  A closed-form solution for multilinear PARAFAC decompositions , 2008, 2008 5th IEEE Sensor Array and Multichannel Signal Processing Workshop.

[23]  R. Bro,et al.  Practical aspects of PARAFAC modeling of fluorescence excitation‐emission data , 2003 .

[24]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[25]  Tamara G. Kolda,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2010, TKDD.

[26]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[27]  Andrzej Cichocki,et al.  Fast damped gauss-newton algorithm for sparse and nonnegative tensor factorization , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[29]  Daniel Graupe,et al.  Topographic component (Parallel Factor) analysis of multichannel evoked potentials: Practical issues in trilinear spatiotemporal decomposition , 2005, Brain Topography.

[30]  Zbynek Koldovský,et al.  Simultaneous search for all modes in multilinear models , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[31]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[32]  Nicholas D. Sidiropoulos,et al.  Parafac techniques for signal separation , 2000 .