Jacobi-type algorithms for homogeneous polynomial optimization on Stiefel manifolds with applications to tensor approximations

This paper mainly studies the gradient-based Jacobi-type algorithms to maximize two classes of homogeneous polynomials with orthogonality constraints, and establish their convergence properties. For the first class of homogeneous polynomials subject to a constraint on a Stiefel manifold, we reformulate it as an optimization problem on a unitary group, which makes it possible to apply the gradient-based Jacobi-type (Jacobi-G) algorithm. Then, if the subproblem can always be represented as a quadratic form, we establish the global convergence of Jacobi-G under any one of three conditions. The convergence result for the first condition is an easy extension of the result in [Usevich et al. SIOPT 2020], while other two conditions are new ones. This algorithm and the convergence properties apply to the well-known joint approximate symmetric tensor diagonalization. For the second class of homogeneous polynomials subject to constraints on the product of Stiefel manifolds, we reformulate it as an optimization problem on the product of unitary groups, and then develop a new gradient-based multi-block Jacobi-type (Jacobi-MG) algorithm to solve it. We establish the global convergence of Jacobi-MG under any one of the above three conditions, if the subproblem can always be represented as a quadratic form. This algorithm and the convergence properties are suitable to the well-known joint approximate tensor diagonalization. As the proximal variants of Jacobi-G and Jacobi-MG, we also propose the Jacobi-GP and Jacobi-MGP algorithms, and establish their global convergence without any further condition. Some numerical results are provided indicating the efficiency of the proposed algorithms.

[1]  Nicolas Boumal.  An Introduction to Optimization on Smooth Manifolds , 2023 .

[2]  E. Begović,et al.  Convergence of a Jacobi-type method for the approximate orthogonal tensor diagonalization , 2021, Calcolo.

[3]  Shuzhong Zhang,et al.  Polar Decomposition-based Algorithms on the Product of Stiefel Manifolds with Applications in Tensor Approximation , 2019, Journal of the Operations Research Society of China.

[4]  Jiawang Nie,et al.  Hermitian Tensor Decompositions , 2019, SIAM J. Matrix Anal. Appl..

[5]  P. Comon,et al.  On the convergence of Jacobi-type algorithms for Independent Component Analysis , 2019, 2020 IEEE 11th Sensor Array and Multichannel Signal Processing Workshop (SAM).

[6]  Sheng-Long Hu,et al.  Linear convergence of an alternating polar decomposition method for low rank orthogonal tensor approximations , 2019, Mathematical Programming.

[7]  Yuning Yang,et al.  The Epsilon-Alternating Least Squares for Orthogonal Low-Rank Tensor Approximation and Its Global Convergence , 2019, SIAM J. Matrix Anal. Appl..

[8]  P. Comon,et al.  Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis , 2019, ArXiv.

[9]  Shenglong Hu,et al.  An inexact augmented Lagrangian method for computing strongly orthogonal decompositions of tensors , 2019, Computational Optimization and Applications.

[10]  Pierre Comon,et al.  Approximate Matrix and Tensor Diagonalization by Unitary Transformations: Convergence of Jacobi-Type Algorithms , 2019, SIAM J. Optim..

[11]  Delin Chu,et al.  Numerical Computation for Orthogonal Low-Rank Approximation of Tensors , 2019, SIAM J. Matrix Anal. Appl..

[12]  Yimin Wei,et al.  Randomized algorithms for the approximations of Tucker and the tensor train decompositions , 2018, Advances in Computational Mathematics.

[13]  P. Comon,et al.  On approximate diagonalization of third order symmetric tensors by orthogonal transformations , 2018, Linear Algebra and its Applications.

[14]  Hans De Sterck,et al.  Nonlinearly preconditioned L‐BFGS as an acceleration mechanism for alternating least squares with application to tensor decomposition , 2018, Numer. Linear Algebra Appl..

[15]  Xiaojun Chen,et al.  A New First-Order Algorithmic Framework for Optimization Problems with Orthogonality Constraints , 2018, SIAM J. Optim..

[16]  Pierre Comon,et al.  Globally convergent Jacobi-type algorithms for simultaneous orthogonal symmetric tensor diagonalization , 2017, SIAM J. Matrix Anal. Appl..

[17]  Nikos D. Sidiropoulos,et al.  Tensor Decomposition for Signal Processing and Machine Learning , 2016, IEEE Transactions on Signal Processing.

[18]  Bo Yu,et al.  Orthogonal Low Rank Tensor Approximation: Alternating Least Squares Method and Its Global Convergence , 2015, SIAM J. Matrix Anal. Appl..

[19]  A. Uschmajew,et al.  A new convergence proof for the higher-order power method and generalizations , 2014, 1407.4586.

[20]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[21]  Reinhold Schneider,et al.  Convergence Results for Projected Line-Search Methods on Varieties of Low-Rank Matrices Via Łojasiewicz Inequality , 2014, SIAM J. Optim..

[22]  Rongjie Lai,et al.  A Splitting Method for Orthogonality Constrained Problems , 2014, J. Sci. Comput..

[23]  P. Absil,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[24]  Paul Van Dooren,et al.  Jacobi Algorithm for the Best Low Multilinear Rank Approximation of Symmetric Tensors , 2013, SIAM J. Matrix Anal. Appl..

[25]  Bo Jiang,et al.  A framework of constraint preserving update schemes for optimization on Stiefel manifold , 2013, Math. Program..

[26]  Pierre Comon,et al.  Canonical Polyadic Decomposition with a Columnwise Orthonormal Factor Matrix , 2012, SIAM J. Matrix Anal. Appl..

[27]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[28]  W. Yin,et al.  A feasible method for optimization with orthogonality constraints , 2012, Mathematical Programming.

[29]  Zhening Li,et al.  Approximation Methods for Polynomial Optimization: Models, Algorithms, and Applications , 2012 .

[30]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[31]  Shuzhong Zhang,et al.  Approximation algorithms for homogeneous polynomial optimization with quadratic constraints , 2010, Math. Program..

[32]  Berkant Savas,et al.  Krylov-Type Methods for Tensor Computations , 2010, 1005.0683.

[33]  Visa Koivunen,et al.  Conjugate gradient algorithm for optimization under unitary matrix constraint , 2009, Signal Process..

[34]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[35]  Chen Ling,et al.  Biquadratic Optimization Over Unit Spheres and Semidefinite Programming Relaxations , 2009, SIAM J. Optim..

[36]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[37]  Berkant Savas,et al.  A Newton-Grassmann Method for Computing the Best Multilinear Rank-(r1, r2, r3) Approximation of a Tensor , 2009, SIAM J. Matrix Anal. Appl..

[38]  Yousef Saad,et al.  On the Tensor SVD and the Optimal Low Rank Orthogonal Approximation of Tensors , 2008, SIAM J. Matrix Anal. Appl..

[39]  Carla D. Moravitz Martin,et al.  A Jacobi-Type Method for Computing Orthogonal Tensor Decompositions , 2008, SIAM J. Matrix Anal. Appl..

[40]  Visa Koivunen,et al.  Steepest Descent Algorithms for Optimization Under Unitary Matrix Constraint , 2008, IEEE Transactions on Signal Processing.

[41]  Pierre-Antoine Absil,et al.  Trust-Region Methods on Riemannian Manifolds , 2007, Found. Comput. Math..

[42]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[43]  Robert E. Mahony,et al.  Convergence of the Iterates of Descent Methods for Analytic Cost Functions , 2005, SIAM J. Optim..

[44]  P. Comon.  Contrasts, independent component analysis, and blind deconvolution , 2004 .

[45]  Athina P. Petropulu,et al.  Joint singular value decomposition - a new tool for separable representation of images , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[46]  Joos Vandewalle,et al.  Independent component analysis and (simultaneous) third-order tensor diagonalization , 2001, IEEE Trans. Signal Process..

[47]  E. Oja,et al.  Independent Component Analysis , 2001 .

[48]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..

[49]  Haesun Park,et al.  A Procrustes problem on the Stiefel manifold , 1999, Numerische Mathematik.

[50]  A. Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[51]  P. Comon.  Tensor Diagonalization, A useful Tool in Signal Processing , 1994 .

[52]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[53]  W. Gander,et al.  A Constrained Eigenvalue Problem , 1989 .

[54]  B. A. D. H. Brandwood.  A complex gradient operator and its applica-tion in adaptive array theory , 1983 .

[55]  Zhou Sheng,et al.  Jacobi-type algorithms for homogeneous polynomial optimization on Stiefel manifolds with applications to tensor approximations , 2023, Math. Comput..

[56]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[57]  Zhi-Quan Luo,et al.  A Semidefinite Relaxation Scheme for Multivariate Quartic Polynomial Optimization with Quadratic Constraints , 2010, SIAM J. Optim..

[58]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[59]  P. Comon,et al.  TENSOR DIAGONALIZATION BY ORTHOGONAL TRANSFORMS , 2007 .

[60]  N. Levenberg,et al.  Function theory in several complex variables , 2001 .

[61]  Jonathan H. Manton,et al.  Modified steepest descent and Newton algorithms for orthogonally constrained optimisation. Part I. The complex Stiefel manifold , 2001, Proceedings of the Sixth International Symposium on Signal Processing and its Applications (Cat.No.01EX467).

[62]  Joos Vandewalle,et al.  Blind source separation by simultaneous third-order tensor diagonalization , 1996, 1996 8th European Signal Processing Conference (EUSIPCO 1996).

[63]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[64]  Harold R. Parks,et al.  A Primer of Real Analytic Functions , 1992 .

[65]  Signal Processing , 1991 .

[66]  S. Krantz.  Function theory of several complex variables , 1982 .

[67]  S. Łojasiewicz.  Ensembles semi-analytiques , 1965 .