CANDECOMP/PARAFAC Decomposition of High-Order Tensors Through Tensor Reshaping

In general, algorithms for order-3 CANDECOMP/ PARAFAC (CP), also coined canonical polyadic decomposition (CPD), are easy to implement and can be extended to higher order CPD. Unfortunately, the algorithms become computationally demanding, and they are often not applicable to higher order and relatively large scale tensors. In this paper, by exploiting the uniqueness of CPD and the relation of a tensor in Kruskal form and its unfolded tensor, we propose a fast approach to deal with this problem. Instead of directly factorizing the high order data tensor, the method decomposes an unfolded tensor with lower order, e.g., order-3 tensor. On the basis of the order-3 estimated tensor, a structured Kruskal tensor, of the same dimension as the data tensor, is then generated, and decomposed to find the final solution using fast algorithms for the structured CPD. In addition, strategies to unfold tensors are suggested and practically verified in the paper.

[1]  F. L. Hitchcock.  Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[2]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[3]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[4]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[5]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[6]  B. Kowalski,et al.  Tensorial resolution: A direct trilinear decomposition , 1990 .

[7]  A. Franc.  Etude algébrique des multitableaux : apports de l'algèbre tensorielle , 1992 .

[8]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[9]  Ben C. Mitchell,et al.  Slowly converging parafac sequences: Swamps and two‐factor degeneracies , 1994 .

[10]  B. Kowalski,et al.  Error analysis of the generalized rank annihilation method , 1994 .

[11]  J. Pernier,et al.  Stimulus Specificity of Phase-Locked and Non-Phase-Locked 40 Hz Visual Responses in Human , 1996, The Journal of Neuroscience.

[12]  R. Harshman,et al.  Relating two proposed methods for speedup of algorithms for fitting two- and three-way principal component and related multilinear models , 1997 .

[13]  P. Paatero.  A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis , 1997 .

[14]  P. Paatero,et al.  THREE-WAY (PARAFAC) FACTOR ANALYSIS : EXAMINATION AND COMPARISON OF ALTERNATIVE COMPUTATIONAL METHODS AS APPLIED TO ILL-CONDITIONED DATA , 1998 .

[15]  Henk A. L. Kiers,et al.  A three–step algorithm for CANDECOMP/PARAFAC analysis of large data sets with multicollinearity , 1998 .

[16]  Rasmus Bro,et al.  Improving the speed of multi-way algorithms:: Part I. Tucker3 , 1998 .

[17]  Rasmus Bro,et al.  MULTI-WAY ANALYSIS IN THE FOOD INDUSTRY Models, Algorithms & Applications , 1998 .

[18]  Nicholas D. Sidiropoulos,et al.  Parafac techniques for signal separation , 2000 .

[19]  Rasmus Bro,et al.  The N-way Toolbox for MATLAB , 2000 .

[20]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[21]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[22]  N. Sidiropoulos,et al.  On the uniqueness of multilinear decomposition of N‐way arrays , 2000 .

[23]  P. Paatero.  Construction and analysis of degenerate PARAFAC models , 2000 .

[24]  Nikos D. Sidiropoulos,et al.  Cramer-Rao lower bounds for low-rank decomposition of multidimensional arrays , 2001, IEEE Trans. Signal Process..

[25]  Ilghiz Ibraghimov,et al.  Application of the three‐way decomposition for matrix compression , 2002, Numer. Linear Algebra Appl..

[26]  R. Bro,et al.  Practical aspects of PARAFAC modeling of fluorescence excitation‐emission data , 2003 .

[27]  Rasmus Bro,et al.  Multi-way Analysis with Applications in the Chemical Sciences , 2004 .

[28]  Amy Nicole Langville,et al.  A Kronecker product approximate preconditioner for SANs , 2004, Numer. Linear Algebra Appl..

[29]  L. Lathauwer,et al.  On the Best Rank-1 and Rank-( , 2004 .

[30]  William S Rayens,et al.  Structure-seeking multilinear methods for the analysis of fMRI data , 2004, NeuroImage.

[31]  Pierre Comon,et al.  Enhanced Line Search: A Novel Method to Accelerate PARAFAC , 2008, SIAM J. Matrix Anal. Appl..

[32]  Tamir Hazan,et al.  Non-negative tensor factorization with applications to statistics and computer vision , 2005, ICML.

[33]  Boris N. Khoromskij,et al.  Hierarchical Kronecker tensor-product approximations , 2005, J. Num. Math..

[34]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[35]  Lieven De Lathauwer,et al.  A Link between the Canonical Decomposition in Multilinear Algebra and Simultaneous Matrix Diagonalization , 2006, SIAM J. Matrix Anal. Appl..

[36]  Lars Kai Hansen,et al.  Parallel Factor Analysis as an exploratory tool for wavelet transformed event-related EEG , 2006, NeuroImage.

[37]  Lieven De Lathauwer,et al.  Tensor-based techniques for the blind separation of DS-CDMA signals , 2007, Signal Process..

[38]  A. Stegeman,et al.  On Kruskal's uniqueness condition for the Candecomp/Parafac decomposition , 2007 .

[39]  Lars Kai Hansen,et al.  ERPWAVELAB A toolbox for multi-channel analysis of time–frequency transformed event related potentials , 2007, Journal of Neuroscience Methods.

[40]  Boris N. Khoromskij,et al.  Mathematik in den Naturwissenschaften Leipzig Tensor-Product Approximation to Operators and Functions in High Dimensions , 2007 .

[41]  L. De Lathauwer,et al.  Canonical decomposition of even higher order cumulant arrays for blind underdetermined mixture identification , 2008, 2008 5th IEEE Sensor Array and Multichannel Signal Processing Workshop.

[42]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[43]  A. Stegeman,et al.  On the Non-Existence of Optimal Solutions and the Occurrence of “Degeneracy” in the CANDECOMP/PARAFAC Model , 2008, Psychometrika.

[44]  M. Haardt,et al.  A closed-form solution for multilinear PARAFAC decompositions , 2008, 2008 5th IEEE Sensor Array and Multichannel Signal Processing Workshop.

[45]  Michael W. Berry,et al.  Discussion Tracking in Enron Email using PARAFAC. , 2008 .

[46]  Lieven De Lathauwer,et al.  Blind Identification of Underdetermined Mixtures by Simultaneous Matrix Diagonalization , 2008, IEEE Transactions on Signal Processing.

[47]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[48]  Richard A. Harshman,et al.  An efficient algorithm for Parafac with uncorrelated mode‐A components applied to large I × J × K data sets with I >> JK , 2009 .

[49]  Pierre Comon,et al.  Tensors versus matrices usefulness and unexpected properties , 2009, 2009 IEEE/SP 15th Workshop on Statistical Signal Processing.

[50]  F. Chinesta,et al.  Recent advances on the use of separated representations , 2009 .

[51]  A. Stegeman.  Using the simultaneous generalized Schur decomposition as a Candecomp/Parafac algorithm for ill‐conditioned data , 2009 .

[52]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[53]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[54]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[55]  Zbynek Koldovský,et al.  Simultaneous search for all modes in multilinear models , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[56]  Alwin Stegeman,et al.  On Uniqueness of the nth Order Tensor Decomposition into Rank-1 Terms with Linear Independence in One Mode , 2010, SIAM J. Matrix Anal. Appl..

[57]  A. Cichocki,et al.  Tensor decompositions for feature extraction and classification of high dimensional datasets , 2010 .

[58]  P. Constantine,et al.  A Surrogate Accelerated Bayesian Inverse Analysis of the HyShot II Supersonic Combustion Data , 2010 .

[59]  P. Tichavský,et al.  Stability analysis and fast damped-gauss-newton algorithm for INDSCAL tensor decomposition , 2011, 2011 IEEE Statistical Signal Processing Workshop (SSP).

[60]  Pierre Comon,et al.  Blind Identification of Underdetermined Mixtures Based on the Characteristic Function: The Complex Case , 2011, IEEE Transactions on Signal Processing.

[61]  Andrzej Cichocki,et al.  Fast damped gauss-newton algorithm for sparse and nonnegative tensor factorization , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[62]  Daniel M. Dunlavy,et al.  A scalable optimization approach for fitting canonical tensor decompositions , 2011 .

[63]  Jorge N. Tendeiro,et al.  Some New Results on Orthogonally Constrained Candecomp , 2011, J. Classif..

[64]  Tamara G. Kolda,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2010, TKDD.

[65]  Pierre Comon,et al.  Computing the polyadic decomposition of nonnegative third order tensors , 2011, Signal Process..

[66]  Zbynek Koldovský,et al.  Weight Adjusted Tensor Method for Blind Separation of Underdetermined Mixtures of Nonstationary Sources , 2011, IEEE Transactions on Signal Processing.

[67]  Alwin Stegeman,et al.  On Uniqueness of the Canonical Tensor Decomposition with Some Form of Symmetry , 2011, SIAM J. Matrix Anal. Appl..

[68]  Liqun Qi,et al.  New ALS Methods With Extrapolating Search Directions and Optimal Step Size for Complex-Valued Tensor Decompositions , 2011, IEEE Transactions on Signal Processing.

[69]  Zbynek Koldovský,et al.  Stability of CANDECOMP-PARAFAC tensor decomposition , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[70]  L. Lathauwer,et al.  Canonical Polyadic Decomposition with Orthogonality Constraints , 2012 .

[71]  Pierre Comon,et al.  CONFAC Decomposition Approach to Blind Identification of Underdetermined Mixtures Based on Generating Function Derivatives , 2012, IEEE Transactions on Signal Processing.

[72]  Eugene E. Tyrtyshnikov,et al.  Fast truncation of mode ranks for bilinear tensor operations , 2012, Numer. Linear Algebra Appl..

[73]  A. Atsawarungruangkit,et al.  Generating Correlation Matrices Based on the Boundaries of Their Coefficients , 2012, PloS one.

[74]  Alwin Stegeman,et al.  Candecomp/Parafac: From Diverging Components to a Decomposition in Block Terms , 2012, SIAM J. Matrix Anal. Appl..

[75]  Charles Van Loan,et al.  Block Tensor Unfoldings , 2011, SIAM J. Matrix Anal. Appl..

[76]  Andrzej Cichocki,et al.  On Fast Computation of Gradients for CANDECOMP/PARAFAC Algorithms , 2012, ArXiv.

[77]  Laurent Albera,et al.  Multi-way space-time-wave-vector analysis for EEG source separation , 2012, Signal Process..

[78]  Pierre Comon,et al.  Canonical Polyadic Decomposition with a Columnwise Orthonormal Factor Matrix , 2012, SIAM J. Matrix Anal. Appl..

[79]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[80]  Zbynek Koldovský,et al.  Cramér-Rao-Induced Bounds for CANDECOMP/PARAFAC Tensor Decomposition , 2012, IEEE Transactions on Signal Processing.

[81]  Andrzej Cichocki,et al.  A further improvement of a fast damped Gauss-Newton algorithm for candecomp-parafac tensor decomposition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[82]  Andrzej Cichocki,et al.  Fast Alternating LS Algorithms for High Order CANDECOMP/PARAFAC Tensor Factorizations , 2013, IEEE Transactions on Signal Processing.

[83]  Pierre Comon,et al.  Blind Multilinear Identification , 2012, IEEE Transactions on Information Theory.