Nonlinearly Preconditioned Optimization on Grassmann Manifolds for Computing Approximate Tucker Tensor Decompositions

Two accelerated optimization algorithms are presented for computing approximate Tucker tensor decompositions, formulated using orthonormal factor matrices, by minimizing error as measured by the Frobenius norm. The first is a nonlinearly preconditioned conjugate gradient (NPCG) algorithm, wherein a nonlinear preconditioner is used to generate a direction which replaces the gradient in the nonlinear conjugate gradient iteration. The second is a nonlinear GMRES (N-GMRES) algorithm, in which a linear combination of past iterates and a tentative new iterate, generated by a nonlinear preconditioner, is minimized to produce an improved search direction. The Euclidean versions of these methods are extended to the manifold setting, where optimization on Grassmann manifolds is used to handle orthonormality constraints and to allow isolated minimizers. Several modifications are required for use on manifolds: logarithmic maps are used to determine required tangent vectors, retraction mappings are used in the line se...

[1]  Sabine Van Huffel,et al.  Best Low Multilinear Rank Approximation of Higher-Order Tensors, Based on the Riemannian Trust-Region Scheme , 2011, SIAM J. Matrix Anal. Appl..

[2]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[3]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[4]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[5]  Reinhold Schneider,et al.  An analysis for the DIIS acceleration method used in quantum chemistry calculations , 2011 .

[6]  Hans De Sterck,et al.  A nonlinearly preconditioned conjugate gradient algorithm for rank‐R canonical tensor approximation , 2014, Numer. Linear Algebra Appl..

[7]  Tamara G. Kolda,et al.  Scalable Tensor Decompositions for Multi-aspect Data Mining , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[8]  Chris H. Q. Ding,et al.  Robust Tucker Tensor Decomposition for Effective Image Representation , 2013, 2013 IEEE International Conference on Computer Vision.

[9]  William W. Hager,et al.  A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search , 2005, SIAM J. Optim..

[10]  Mariya Ishteva.  Numerical Methods for the Best Low Multilinear Rank Approximation of Higher-Order Tensors (Numerieke methoden voor de beste lage multilineaire rang benadering van hogere-orde tensoren) , 2009 .

[11]  Cornelis W. Oosterlee,et al.  KRYLOV SUBSPACE ACCELERATION FOR NONLINEAR MULTIGRID SCHEMES , 1997 .

[12]  Shuzhong Zhang,et al.  On optimal low rank Tucker approximation for tensors: the case for an adjustable core size , 2015, J. Glob. Optim..

[13]  Wolfgang Hackbusch,et al.  An Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples , 2011, Comput. Methods Appl. Math..

[14]  Matthew G. Knepley,et al.  Composing Scalable Nonlinear Algebraic Solvers , 2015, SIAM Rev..

[15]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[16]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[17]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[18]  Pierre-Antoine Absil,et al.  Trust-Region Methods on Riemannian Manifolds , 2007, Found. Comput. Math..

[19]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[20]  Nicolas Boumal,et al.  Riemannian Trust Regions with Finite-Difference Hessian Approximations are Globally Convergent , 2015, GSI.

[21]  Rasmus Bro,et al.  MULTI-WAY ANALYSIS IN THE FOOD INDUSTRY Models, Algorithms & Applications , 1998 .

[22]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[23]  Sabine Van Huffel,et al.  Differential-geometric Newton method for the best rank-(R1, R2, R3) approximation of tensors , 2008, Numerical Algorithms.

[24]  L. Lathauwer,et al.  On the best low multilinear rank approximation of higher-order tensors , 2010 .

[25]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[26]  K. Meerbergen,et al.  On the truncated multilinear singular value decomposition , 2011 .

[27]  Lars Grasedyck,et al.  F ¨ Ur Mathematik in Den Naturwissenschaften Leipzig a Projection Method to Solve Linear Systems in Tensor Format a Projection Method to Solve Linear Systems in Tensor Format , 2022 .

[28]  Christina Freytag.  Geometry Of Pdes And Mechanics , 2016 .

[29]  Hans De Sterck.  NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS Numer. Linear Algebra Appl. (2012) Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/nla.1837 Steepest descent preconditioning for nonlinear , 2022 .

[30]  J. Daniel,et al.  A conjugate gradient approach to nonlinear elliptic boundary value problems in irregular regions , 1974 .

[31]  Berkant Savas,et al.  Quasi-Newton Methods on Grassmannians and Multilinear Approximations of Tensors , 2009, SIAM J. Sci. Comput..

[32]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[33]  Tamara G. Kolda,et al.  Poblano v1.0: A Matlab Toolbox for Gradient-Based Optimization , 2010 .

[34]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[35]  Hans De Sterck,et al.  A Nonlinear GMRES Optimization Algorithm for Canonical Tensor Decomposition , 2011, SIAM J. Sci. Comput..

[36]  Tamara G. Kolda,et al.  Categories and Subject Descriptors: G.4 [Mathematics of Computing]: Mathematical Software— , 2022 .

[37]  Berkant Savas,et al.  A Newton-Grassmann Method for Computing the Best Multilinear Rank-(r1, r2, r3) Approximation of a Tensor , 2009, SIAM J. Matrix Anal. Appl..

[38]  Homer F. Walker,et al.  Anderson Acceleration for Fixed-Point Iterations , 2011, SIAM J. Numer. Anal..

[39]  Gene H. Golub,et al.  Numerical solution of nonlinear elliptic partial differential equations by a generalized conjugate gradient method , 1976, Computing.

[40]  Donald G. M. Anderson.  Iterative Procedures for Nonlinear Integral Equations , 1965, JACM.

[41]  Raf Vandebril,et al.  A New Truncation Strategy for the Higher-Order Singular Value Decomposition , 2012, SIAM J. Sci. Comput..

[42]  David J. Thuente,et al.  Line search algorithms with guaranteed sufficient decrease , 1994, TOMS.

[43]  Berkant Savas,et al.  Handwritten digit classification using higher order singular value decomposition , 2007, Pattern Recognit..

[44]  W. Hager,et al.  A SURVEY OF NONLINEAR CONJUGATE GRADIENT METHODS , 2005 .

[45]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[46]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[47]  Hans D. Mittelmann,et al.  On the efficient solution of nonlinear finite element equations I , 1980 .

[48]  Yousef Saad,et al.  Two classes of multisecant methods for nonlinear acceleration , 2009, Numer. Linear Algebra Appl..

[49]  David E. Booth,et al.  Multi-Way Analysis: Applications in the Chemical Sciences , 2005, Technometrics.

[50]  N. T. Son.  A real time procedure for affinely dependent parametric model order reduction using interpolation on Grassmann manifolds , 2013 .

[51]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[52]  Joos Vandewalle,et al.  On the Best Rank-1 and Rank-(R1 , R2, ... , RN) Approximation of Higher-Order Tensors , 2000, SIAM J. Matrix Anal. Appl..