Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors

In this paper we proposed quasi-Newton and limited memory quasi-Newton methods for objective functions defined on Grassmannians or a product of Grassmannians. Specifically we defined BFGS and limited memory BFGS updates in local and global coordinates on Grassmannians or a product of these. We proved that, when local coordinates are used, our BFGS updates on Grassmannians share the same optimality property as the usual BFGS updates on Euclidean spaces. When applied to the best multilinear rank approximation problem for general and symmetric tensors, our approach yields fast, robust, and accurate algorithms that exploit the special Grassmannian structure of the respective problems and which work on tensors of large dimensions and arbitrarily high order. Extensive numerical experiments are included to substantiate our claims.

[1]  Joos Vandewalle,et al.  Dimensionality Reduction in ICA and Rank-(R1, R2, ..., RN) Reduction in Multilinear Algebra , 2004, ICA.

[2]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[3]  S. H. Cheng,et al.  A Modified Cholesky Algorithm Based on a Symmetric Indefinite Factorization , 1998, SIAM J. Matrix Anal. Appl..

[4]  Joos Vandewalle,et al.  A Grassmann-Rayleigh Quotient Iteration for Dimensionality Reduction in ICA , 2004, ICA.

[5]  F. L. Hitchcock.  The Expression of a Tensor or a Polyadic as a Sum of Products , 1927 .

[6]  Lennart Simonsson.  Subspace Computations via Matrix Decompositions and Geometric Optimization , 2006 .

[7]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[8]  R. Tennant.  Algebra , 1941, Nature.

[9]  T. Yokonuma.  Tensor Spaces and Exterior Algebra , 1992 .

[10]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[11]  R. Schnabel,et al.  A NEW DERIVATION OF SYMMETRIC POSITIVE DEFINITE SECANT UPDATES , 1980 .

[12]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[13]  Pierre-Antoine Absil,et al.  Trust-Region Methods on Riemannian Manifolds , 2007, Found. Comput. Math..

[14]  L. Lathauwer,et al.  Dimensionality reduction for higher-order tensors: algorithms and applications , 2008 .

[15]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[16]  Pierre Comon,et al.  Blind identification of under-determined mixtures based on the characteristic function , 2006, Signal Process..

[17]  Ganesh R. Naik,et al.  Introduction: Independent Component Analysis , 2012 .

[18]  Eva Lundstr.  Adaptive Eigenvalue Computations Using Newton's Method on the Grassmann Manifold , 1999 .

[19]  Pierre Comon,et al.  Blind identification of under-determined mixtures based on the characteristic function , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[20]  J. J. Moré,et al.  Quasi-Newton Methods, Motivation and Theory , 1974 .

[21]  W. Boothby.  An introduction to differentiable manifolds and Riemannian geometry , 1975 .

[22]  B. Khoromskij,et al.  Low rank Tucker-type tensor approximation to classical potentials , 2007 .

[23]  L. Lathauwer,et al.  Dimensionality reduction in higher-order signal processing and rank-(R1,R2,…,RN) reduction in multilinear algebra , 2004 .

[24]  Pierre Comon,et al.  Decomposition of quantics in sums of powers of linear forms , 1996, Signal Process..

[25]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[26]  Lars Kai Hansen,et al.  ERPWAVELAB A toolbox for multi-channel analysis of time–frequency transformed event related potentials , 2007, Journal of Neuroscience Methods.

[27]  Jorge Nocedal,et al.  Representations of quasi-Newton matrices and their use in limited memory methods , 1994, Math. Program..

[28]  S. D. Cutkosky,et al.  Multilinear Algebra , 2019, Differential Forms.

[29]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[30]  Boris N. Khoromskij,et al.  Verification of the cross 3D algorithm on quantum chemistry data , 2008 .

[31]  H. Walker.  Quasi-Newton Methods , 1978 .

[32]  Lieven De Lathauwer.  Tucker compression, parallel factor analysis and block term decompositions : new results , 2010 .

[33]  F. L. Hitchcock.  Multiple Invariants and Generalized Rank of a P‐Way Matrix or Tensor , 1928 .

[34]  Gene H. Golub,et al.  Genericity And Rank Deficiency Of High Order Symmetric Tensors , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[35]  D. Gabay.  Minimizing a differentiable function over a differential manifold , 1982 .

[36]  L. Lathauwer,et al.  On the best low multilinear rank approximation of higher-order tensors , 2010 .

[37]  Y. Wong,et al.  Differentiable Manifolds , 2009 .

[38]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[39]  Eugene E. Tyrtyshnikov,et al.  Tucker Dimensionality Reduction of Three-Dimensional Arrays in Linear Time , 2008, SIAM J. Matrix Anal. Appl..

[40]  Tamara G. Kolda,et al.  Categories and Subject Descriptors: G.4 [Mathematics of Computing]: Mathematical Software— , 2022 .

[41]  Berkant Savas,et al.  A Newton-Grassmann Method for Computing the Best Multilinear Rank-(r1, r2, r3) Approximation of a Tensor , 2009, SIAM J. Matrix Anal. Appl..

[42]  Boris N. Khoromskij,et al.  Multigrid Accelerated Tensor Approximation of Function Related Multidimensional Arrays , 2009, SIAM J. Sci. Comput..

[43]  Gene H. Golub,et al.  Symmetric Tensors and Symmetric Tensor Rank , 2008, SIAM J. Matrix Anal. Appl..

[44]  G. Golub,et al.  A tensor higher-order singular value decomposition for integrative analysis of DNA microarray data from different studies , 2007, Proceedings of the National Academy of Sciences.

[45]  L. Lathauwer,et al.  Signal Processing based on Multilinear Algebra , 1997 .

[46]  Ed Anderson,et al.  LAPACK Users' Guide , 1995 .

[47]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[48]  Berkant Savas.  Toolbox for Grassmann Manifold Computations , 2008 .

[49]  D. Goldfarb.  Factorized variable metric methods for unconstrained optimization , 1976 .

[50]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[51]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[52]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[53]  Demetri Terzopoulos,et al.  Multilinear subspace analysis of image ensembles , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[54]  Phillip A. Regalia,et al.  On the Best Rank-1 Approximation of Higher-Order Supersymmetric Tensors , 2001, SIAM J. Matrix Anal. Appl..

[55]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..