Optimization-Based Algorithms for Tensor Decompositions: Canonical Polyadic Decomposition, Decomposition in Rank-(Lr, Lr, 1) Terms, and a New Generalization

The canonical polyadic and rank-$(L_r,L_r,1)$ block term decomposition (CPD and BTD, respectively) are two closely related tensor decompositions. The CPD and, recently, BTD are important tools in psychometrics, chemometrics, neuroscience, and signal processing. We present a decomposition that generalizes these two and develop algorithms for its computation. Among these algorithms are alternating least squares schemes, several general unconstrained optimization techniques, and matrix-free nonlinear least squares methods. In the latter we exploit the structure of the Jacobian's Gramian to reduce computational and memory cost. Combined with an effective preconditioner, numerical experiments confirm that these methods are among the most efficient and robust currently available for computing the CPD, rank-$(L_r,L_r,1)$ BTD, and their generalized decomposition.

[1]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part II: Definitions and Uniqueness , 2008, SIAM J. Matrix Anal. Appl..

[2]  Richard H. Byrd,et al.  Approximate solution of the trust region problem by minimization over two-dimensional subspaces , 1988, Math. Program..

[3]  Chikio Hayashi,et al.  A NEW ALGORITHM TO SOLVE PARAFAC-MODEL , 1982 .

[4]  P. Paatero.  Construction and analysis of degenerate PARAFAC models , 2000 .

[5]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[6]  Lieven De Lathauwer,et al.  On the Uniqueness of the Canonical Polyadic Decomposition of Third-Order Tensors - Part I: Basic Results and Uniqueness of One Factor Matrix , 2013, SIAM J. Matrix Anal. Appl..

[7]  F. L. Hitchcock.  The Expression of a Tensor or a Polyadic as a Sum of Products , 1927 .

[8]  Lieven De Lathauwer,et al.  On the Uniqueness of the Canonical Polyadic Decomposition of Third-Order Tensors - Part II: Uniqueness of the Overall Decomposition , 2013, SIAM J. Matrix Anal. Appl..

[9]  Mike E. Davies,et al.  Latent Variable Analysis and Signal Separation , 2010 .

[10]  Rasmus Bro,et al.  Multi-way Analysis with Applications in the Chemical Sciences , 2004 .

[11]  P. Paatero.  A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis , 1997 .

[12]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[13]  J. Kruskal,et al.  How 3-MFA data can cause degenerate parafac solutions, among other relationships , 1989 .

[14]  R. Remmert,et al.  Theory of Complex Functions , 1990 .

[15]  N.D. Sidiropoulos,et al.  Blind multiuser detection in W-CDMA systems with large delay spread , 2001, IEEE Signal Processing Letters.

[16]  Lieven De Lathauwer,et al.  A Link between the Canonical Decomposition in Multilinear Algebra and Simultaneous Matrix Diagonalization , 2006, SIAM J. Matrix Anal. Appl..

[17]  Lieven De Lathauwer,et al.  An enhanced line search scheme for complex-valued tensor decompositions. Application in DS-CDMA , 2008, Signal Process..

[18]  Lieven De Lathauwer,et al.  Blind Separation of Exponential Polynomials and the Decomposition of a Tensor in Rank-(Lr, Lr, 1) Terms , 2011, SIAM J. Matrix Anal. Appl..

[19]  Pierre Comon,et al.  Enhanced Line Search: A Novel Method to Accelerate PARAFAC , 2008, SIAM J. Matrix Anal. Appl..

[20]  Daniel M. Dunlavy,et al.  An Optimization Approach for Fitting Canonical Tensor Decompositions. , 2009 .

[21]  P. Paatero.  The Multilinear Engine—A Table-Driven, Least Squares Program for Solving Multilinear Problems, Including the n-Way Parallel Factor Analysis Model , 1999 .

[22]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part III: Alternating Least Squares Algorithms , 2008, SIAM J. Matrix Anal. Appl..

[23]  J. Dennis,et al.  Two new unconstrained optimization algorithms which use function and gradient values , 1979 .

[24]  A. Bos.  Complex gradient and Hessian , 1994 .

[25]  Lieven De Lathauwer,et al.  Unconstrained Optimization of Real Functions in Complex Variables , 2012, SIAM J. Optim..

[26]  Jerry M. Mendel,et al.  The constrained total least squares technique and its applications to harmonic superresolution , 1991, IEEE Trans. Signal Process..

[27]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[28]  A. Stegeman,et al.  On the Non-Existence of Optimal Solutions and the Occurrence of “Degeneracy” in the CANDECOMP/PARAFAC Model , 2008, Psychometrika.

[29]  Alwin Stegeman,et al.  Low-Rank Approximation of Generic p˟q˟2 Arrays and Diverging Components in the Candecomp/Parafac Model , 2008, SIAM J. Matrix Anal. Appl..

[30]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[31]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[32]  Lieven De Lathauwer,et al.  Swamp reducing technique for tensor decomposition , 2008, 2008 16th European Signal Processing Conference.

[33]  Nikos D. Sidiropoulos,et al.  Blind PARAFAC receivers for DS-CDMA systems , 2000, IEEE Trans. Signal Process..

[34]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[35]  Lieven De Lathauwer,et al.  Block Component Analysis, a New Concept for Blind Source Separation , 2012, LVA/ICA.

[36]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[37]  L. Imhof.  Matrix Algebra and Its Applications to Statistics and Econometrics , 1998 .

[38]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[39]  Lieven De Lathauwer,et al.  Decompositions of a Higher-Order Tensor in Block Terms - Part I: Lemmas for Partitioned Matrices , 2008, SIAM J. Matrix Anal. Appl..

[40]  Shirley Dex,et al.  JR 旅客販売総合システム（マルス）における運用及び管理について , 1991 .

[41]  Nikos D. Sidiropoulos,et al.  Kruskal's permutation lemma and the identification of CANDECOMP/PARAFAC and bilinear models with constant modulus constraints , 2004, IEEE Transactions on Signal Processing.

[42]  P. Kroonenberg.  Applied Multiway Data Analysis , 2008 .

[43]  F. L. Hitchcock.  Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[44]  Andrzej Cichocki,et al.  Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC , 2012, SIAM J. Matrix Anal. Appl..

[45]  Lieven De Lathauwer,et al.  Blind Deconvolution of DS-CDMA Signals by Means of Decomposition in Rank-$(1,L,L)$ Terms , 2008, IEEE Transactions on Signal Processing.

[46]  L. Lathauwer,et al.  An enhanced plane search scheme for complex-valued tensor decompositions , 2010 .

[47]  Rasmus Bro,et al.  MULTI-WAY ANALYSIS IN THE FOOD INDUSTRY Models, Algorithms & Applications , 1998 .

[48]  Ben C. Mitchell,et al.  Slowly converging parafac sequences: Swamps and two‐factor degeneracies , 1994 .

[49]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[50]  Pierre Comon,et al.  Special Issue on Tensor Decompositions and Applications , 2008, SIAM J. Matrix Anal. Appl..

[51]  N. Sidiropoulos,et al.  On the uniqueness of multilinear decomposition of N‐way arrays , 2000 .

[52]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[53]  J. Kruskal.  Rank, decomposition, and uniqueness for 3-way and n -way arrays , 1989 .

[54]  Andrzej Cichocki,et al.  On Fast Computation of Gradients for CANDECOMP/PARAFAC Algorithms , 2012, ArXiv.

[55]  Rasmus Bro,et al.  A comparison of algorithms for fitting the PARAFAC model , 2006, Comput. Stat. Data Anal..

[56]  Lieven De Lathauwer,et al.  A short introduction to tensor-based methods for Factor Analysis and Blind Source Separation , 2011, 2011 7th International Symposium on Image and Signal Processing and Analysis (ISPA).

[57]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[58]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[59]  M. Powell.  A New Algorithm for Unconstrained Optimization , 1970 .

[60]  A. Stegeman.  Degeneracy in Candecomp/Parafac explained for p × p × 2 arrays of rank p + 1 or higher , 2006 .

[61]  Moritz Diehl,et al.  The Lifted Newton Method and Its Application in Optimization , 2009, SIAM J. Optim..

[62]  Lieven De Lathauwer,et al.  Block component analysis , 2011 .

[63]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[64]  Nikos D. Sidiropoulos,et al.  Parallel factor analysis in sensor array processing , 2000, IEEE Trans. Signal Process..

[65]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[66]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[67]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[68]  Tamara G. Kolda,et al.  Efficient MATLAB Computations with Sparse and Factored Tensors , 2007, SIAM J. Sci. Comput..