Jacobi-type algorithms for homogeneous polynomial optimization on Stiefel manifolds with applications to tensor approximations

In this paper, we mainly study the gradient based Jacobi-type algorithms to maximize two classes of homogeneous polynomials with orthogonality constraints, and establish their convergence properties. For the first class of homogeneous polynomials subject to a constraint on a Stiefel manifold, we reformulate it as an optimization problem on a unitary group, which makes it possible to apply the gradient based Jacobi-type (Jacobi-G) algorithm. Then, if the subproblem can be always represented as a quadratic form, we establish the global convergence of Jacobi-G under any one of the three conditions (A1), (A2) and (A3). The convergence result for (A1) is an easy extension of the result in [Usevich et al. SIOPT 2020], while (A2) and (A3) are two new ones. This algorithm and the convergence properties apply to the well-known joint approximate symmetric tensor diagonalization and joint approximate symmetric trace maximization. For the second class of homogeneous polynomials subject to constraints on the product of Stiefel manifolds, we similarly reformulate it as an optimization problem on the product of unitary groups, and then develop a new gradient based multi-block Jacobi-type (Jacobi-MG) algorithm to solve it. We similarly establish the global convergence of Jacobi-MG under any one of the three conditions (A1), (A2) and (A3), if the subproblem can be always represented as a quadratic form. This algorithm and the convergence properties apply to the well-known joint approximate tensor diagonalization and joint approximate tensor compression. As the proximal variants of Jacobi-G and Jacobi-MG, we also propose the Jacobi-GP and Jacobi-MGP algorithms, and establish their global convergence without any further condition. Some numerical results are provided indicating the efficiency of the proposed algorithms. Date: October 6, 2021. 2010 Mathematics Subject Classification. 15A69, 65F99, 90C30.

[1]  Reinhold Schneider,et al.  Convergence Results for Projected Line-Search Methods on Varieties of Low-Rank Matrices Via Łojasiewicz Inequality , 2014, SIAM J. Optim..

[2]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[3]  Jiawang Nie,et al.  Hermitian Tensor Decompositions , 2020, SIAM J. Matrix Anal. Appl..

[4]  Pierre Comon,et al.  Globally convergent Jacobi-type algorithms for simultaneous orthogonal symmetric tensor diagonalization , 2017, SIAM J. Matrix Anal. Appl..

[5]  Hans De Sterck,et al.  Nonlinearly preconditioned L‐BFGS as an acceleration mechanism for alternating least squares with application to tensor decomposition , 2018, Numer. Linear Algebra Appl..

[6]  Robert E. Mahony,et al.  Convergence of the Iterates of Descent Methods for Analytic Cost Functions , 2005, SIAM J. Optim..

[7]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[8]  Berkant Savas,et al.  A Newton-Grassmann Method for Computing the Best Multilinear Rank-(r1, r2, r3) Approximation of a Tensor , 2009, SIAM J. Matrix Anal. Appl..

[9]  B. A. D. H. Brandwood.  A complex gradient operator and its applica-tion in adaptive array theory , 1983 .

[10]  Bo Jiang,et al.  A framework of constraint preserving update schemes for optimization on Stiefel manifold , 2013, Math. Program..

[11]  A. Uschmajew,et al.  A new convergence proof for the higher-order power method and generalizations , 2014, 1407.4586.

[12]  P. Comon.  Tensor Diagonalization, A useful Tool in Signal Processing , 1994 .

[13]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[14]  Erna Begovic.  Convergence of a Jacobi-type method for the approximate orthogonal tensor diagonalization , 2021, ArXiv.

[15]  Bo Yu,et al.  Orthogonal Low Rank Tensor Approximation: Alternating Least Squares Method and Its Global Convergence , 2015, SIAM J. Matrix Anal. Appl..

[16]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[17]  Pierre Comon,et al.  On the convergence of Jacobi-type algorithms for Independent Component Analysis , 2019, 2020 IEEE 11th Sensor Array and Multichannel Signal Processing Workshop (SAM).

[18]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[19]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[20]  Yuning Yang,et al.  The Epsilon-Alternating Least Squares for Orthogonal Low-Rank Tensor Approximation and Its Global Convergence , 2019, SIAM J. Matrix Anal. Appl..

[21]  Sheng-Long Hu,et al.  Linear convergence of an alternating polar decomposition method for low rank orthogonal tensor approximations , 2019, Mathematical Programming.

[22]  P. Comon,et al.  TENSOR DIAGONALIZATION BY ORTHOGONAL TRANSFORMS , 2007 .

[23]  Berkant Savas,et al.  Krylov-Type Methods for Tensor Computations , 2010, 1005.0683.

[24]  Athina P. Petropulu,et al.  Joint singular value decomposition - a new tool for separable representation of images , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[25]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[26]  Pierre Comon,et al.  Canonical Polyadic Decomposition with a Columnwise Orthonormal Factor Matrix , 2012, SIAM J. Matrix Anal. Appl..

[27]  S. Łojasiewicz.  Ensembles semi-analytiques , 1965 .

[28]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[29]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[30]  Shuzhong Zhang,et al.  Polar decomposition based algorithms on the product of Stiefel manifolds with applications in tensor approximation , 2019, ArXiv.

[31]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[32]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[33]  N. Levenberg,et al.  Function theory in several complex variables , 2001 .

[34]  Delin Chu,et al.  Numerical Computation for Orthogonal Low-Rank Approximation of Tensors , 2019, SIAM J. Matrix Anal. Appl..

[35]  Hemant Kumar Pathak,et al.  Function Theory of Several Complex Variables , 2019, Complex Analysis and Applications.

[36]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[37]  Rongjie Lai,et al.  A Splitting Method for Orthogonality Constrained Problems , 2014, J. Sci. Comput..

[38]  Paul Van Dooren,et al.  Jacobi Algorithm for the Best Low Multilinear Rank Approximation of Symmetric Tensors , 2013, SIAM J. Matrix Anal. Appl..

[39]  Carla D. Moravitz Martin,et al.  A Jacobi-Type Method for Computing Orthogonal Tensor Decompositions , 2008, SIAM J. Matrix Anal. Appl..

[40]  Yousef Saad,et al.  On the Tensor SVD and the Optimal Low Rank Orthogonal Approximation of Tensors , 2008, SIAM J. Matrix Anal. Appl..

[41]  P. Comon,et al.  On approximate diagonalization of third order symmetric tensors by orthogonal transformations , 2018, Linear Algebra and its Applications.

[42]  P. Comon.  Contrasts, independent component analysis, and blind deconvolution , 2004 .

[43]  Shenglong Hu,et al.  An inexact augmented Lagrangian method for computing strongly orthogonal decompositions of tensors , 2019, Computational Optimization and Applications.

[44]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[45]  Harold R. Parks,et al.  A Primer of Real Analytic Functions , 1992 .

[46]  Joos Vandewalle,et al.  Independent component analysis and (simultaneous) third-order tensor diagonalization , 2001, IEEE Trans. Signal Process..

[47]  W. Gander,et al.  A Constrained Eigenvalue Problem , 1989 .

[48]  Visa Koivunen,et al.  Conjugate gradient algorithm for optimization under unitary matrix constraint , 2009, Signal Process..

[49]  Nikos D. Sidiropoulos,et al.  Tensor Decomposition for Signal Processing and Machine Learning , 2016, IEEE Transactions on Signal Processing.

[50]  Yimin Wei,et al.  Randomized algorithms for the approximations of Tucker and the tensor train decompositions , 2018, Advances in Computational Mathematics.

[51]  Joos Vandewalle,et al.  Blind source separation by simultaneous third-order tensor diagonalization , 1996, 1996 8th European Signal Processing Conference (EUSIPCO 1996).

[52]  Pierre Comon,et al.  Approximate Matrix and Tensor Diagonalization by Unitary Transformations: Convergence of Jacobi-Type Algorithms , 2019, SIAM J. Optim..

[53]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[54]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[55]  P. Comon,et al.  Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis , 2019, ArXiv.

[56]  Visa Koivunen,et al.  Steepest Descent Algorithms for Optimization Under Unitary Matrix Constraint , 2008, IEEE Transactions on Signal Processing.

[57]  Jonathan H. Manton,et al.  Modified steepest descent and Newton algorithms for orthogonally constrained optimisation. Part I. The complex Stiefel manifold , 2001, Proceedings of the Sixth International Symposium on Signal Processing and its Applications (Cat.No.01EX467).