Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation

A local convergence theorem for calculating canonical low-rank tensor approximations (PARAFAC, CANDECOMP) by the alternating least squares algorithm is established. The main assumption is that the Hessian matrix of the problem is positive definite modulo the scaling indeterminacy. A discussion, whether this is realistic, and numerical illustrations are included. Also regularization is addressed.

[1]  Konstantin Grella,et al.  Sparse tensor spherical harmonics approximation in radiative transfer , 2010 .

[2]  Wang-Q Lim,et al.  Compactly Supported Shearlets , 2010, 1009.4359.

[3]  W. Hackbusch,et al.  Black Box Low Tensor-Rank Approximation Using Fiber-Crosses , 2009 .

[4]  Christian Bender,et al.  Error Criteria for Numerical Solutions of Backward SDEs , 2010 .

[5]  Thorsten Rohwedder,et al.  The continuous Coupled Cluster formulation for the electronic Schrödinger equation , 2013 .

[6]  Klaus Ritter,et al.  Spatial Besov Regularity for Stochastic Partial Differential Equations on Lipschitz Domains , 2010, 1011.1814.

[7]  Wang-Q Lim,et al.  Image Separation Using Shearlets , 2011 .

[8]  Stephan Dahlke,et al.  Piecewise tensor product wavelet bases by extensions and approximation rates , 2013, Math. Comput..

[9]  O. Ernst,et al.  ON THE CONVERGENCE OF GENERALIZED POLYNOMIAL CHAOS EXPANSIONS , 2011 .

[10]  Stephan Dahlke,et al.  A note on quarkonial systems and multilevel partition of unity methods , 2013 .

[11]  Dajana Conte,et al.  Mathematical Modelling and Numerical Analysis an Error Analysis of the Multi-configuration Time-dependent Hartree Method of Quantum Dynamics , 2022 .

[12]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[13]  Wolfgang Dahmen,et al.  Adaptive Petrov-Galerkin Methods for First Order Transport Equations , 2011, SIAM J. Numer. Anal..

[14]  Sadegh Jokar,et al.  Sparse recovery and Kronecker products , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[15]  Erich Novak,et al.  The Curse of Dimensionality for Monotone and Convex Functions of Many Variables , 2010, 1011.3680.

[16]  K. Bredies,et al.  Regularization with non-convex separable constraints , 2009 .

[17]  James C. Bezdek,et al.  Convergence of Alternating Optimization , 2003, Neural Parallel Sci. Comput..

[18]  Lars Grasedyck,et al.  F ¨ Ur Mathematik in Den Naturwissenschaften Leipzig a Projection Method to Solve Linear Systems in Tensor Format a Projection Method to Solve Linear Systems in Tensor Format , 2022 .

[19]  Christian Bender,et al.  Primal and Dual Pricing of Multiple Exercise Options in Continuous Time , 2011, SIAM J. Financial Math..

[20]  Rasmus Bro,et al.  A comparison of algorithms for fitting the PARAFAC model , 2006, Comput. Stat. Data Anal..

[21]  Steffen Dereich,et al.  Multilevel Monte Carlo algorithms for L\'{e}vy-driven SDEs with Gaussian correction , 2011, 1101.1369.

[22]  Erich Novak,et al.  Optimal approximation of elliptic problems by linear and nonlinear mappings IV: Errors in L2 and other norms , 2004, J. Complex..

[23]  Reinhold Schneider,et al.  An analysis for the DIIS acceleration method used in quantum chemistry calculations , 2011 .

[24]  Steffen Dereich,et al.  A multilevel Monte Carlo algorithm for Lévy-driven stochastic differential equations , 2011 .

[25]  P. Maass,et al.  An analysis of electrical impedance tomography with applications to Tikhonov regularization , 2012 .

[26]  Klaus Ritter,et al.  Derandomization of the Euler scheme for scalar stochastic differential equations , 2012, J. Complex..

[27]  Jinchao Xu,et al.  On the Convergence of Iterative Methods for Semidefinite Linear Systems , 2006, SIAM J. Matrix Anal. Appl..

[28]  P. Maass,et al.  Function spaces and optimal currents in impedance tomography , 2011 .

[29]  G. Teschke,et al.  Accelerated projected steepest descent method for nonlinear inverse problems with sparsity constraints , 2010 .

[30]  Lutz Kämmerer,et al.  Interpolation lattices for hyperbolic cross trigonometric polynomials , 2012, J. Complex..

[31]  Lutz Kämmerer,et al.  On the stability of the hyperbolic cross discrete Fourier transform , 2011, Numerische Mathematik.

[32]  Martin J. Mohlenkamp,et al.  Algorithms for Numerical Analysis in High Dimensions , 2005, SIAM J. Sci. Comput..

[33]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[34]  S. Dereich,et al.  Constructive quantization: Approximation by empirical measures , 2011, 1108.5346.

[35]  Gene H. Golub,et al.  Rank-One Approximation to High Order Tensors , 2001, SIAM J. Matrix Anal. Appl..

[36]  G. Kutyniok,et al.  Construction of Compactly Supported Shearlet Frames , 2010, 1003.5481.

[37]  Young-Ju,et al.  CONVERGENCE ANALYSIS ON ITERATIVE METHODS FOR SEMIDEFINITE SYSTEMS , 2008 .

[38]  Winfried Sickel,et al.  Best m-Term Approximation and Sobolev–Besov Spaces of Dominating Mixed Smoothness—the Case of Compact Embeddings , 2012 .

[39]  Stephan Dahlke,et al.  Spatial Besov regularity for semilinear stochastic partial differential equations on bounded Lipschitz domains , 2012, Int. J. Comput. Math..

[40]  Stephan Dahlke,et al.  An adaptive wavelet method for parameter identification problems in parabolic partial differential equations , 2022 .

[41]  Reinhold Schneider,et al.  Error estimates for the Coupled Cluster method , 2013 .

[42]  Lieven De Lathauwer,et al.  Swamp reducing technique for tensor decomposition , 2008, 2008 16th European Signal Processing Conference.

[43]  Elisabeth Ullmann,et al.  Stochastic Galerkin Matrices , 2010, SIAM J. Matrix Anal. Appl..

[44]  Henryk Wozniakowski,et al.  Approximation of infinitely differentiable multivariate functions is intractable , 2009, J. Complex..

[45]  Wolfgang Dahmen,et al.  Adaptivity and variational stabilization for convection-diffusion equations∗ , 2012 .

[46]  Armin Iske,et al.  Curvature analysis of frequency modulated manifolds in dimensionality reduction , 2011 .

[47]  Daniel Rudolf,et al.  Error bounds for computing the expectation by Markov chain Monte Carlo , 2009, Monte Carlo Methods Appl..

[48]  Armin Iske,et al.  Optimal representation of piecewise Hölder smooth bivariate functions by the Easy Path Wavelet Transform , 2013, J. Approx. Theory.

[49]  Ivan Oseledets,et al.  A new tensor decomposition , 2009 .

[50]  J. T. ten Berge,et al.  The link between sufficient conditions by Harshman and by Kruskal for uniqueness in Candecomp/Parafac , 2009 .

[51]  Harry Yserentant,et al.  The mixed regularity of electronic wave functions multiplied by explicit correlation factors , 2011 .

[52]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[53]  Gitta Kutyniok,et al.  Data Separation by Sparse Representations , 2011, Compressed Sensing.

[54]  Tobias Jahnke,et al.  On Reduced Models for the Chemical Master Equation , 2011, Multiscale Model. Simul..

[55]  Howard C. Elman,et al.  Efficient Iterative Solvers for Stochastic Galerkin Discretizations of Log-Transformed Random Diffusion Problems , 2012, SIAM J. Sci. Comput..

[56]  Winfried Sickel,et al.  Best m-term aproximation and tensor product of Sobolev and Besov spaces-the case of non-compact embeddings , 2010 .

[57]  J. Ballani,et al.  Black box approximation of tensors in hierarchical Tucker format , 2013 .

[58]  G. Kutyniok,et al.  Irregular Shearlet Frames: Geometry and Approximation Properties , 2010, 1002.2657.

[59]  Wang-Q Lim,et al.  Compactly supported shearlets are optimally sparse , 2010, J. Approx. Theory.

[60]  Michael Döhler,et al.  Nonequispaced Hyperbolic Cross Fast Fourier Transform , 2010, SIAM J. Numer. Anal..

[61]  Jianwei Ma,et al.  A Review of Curvelets and Recent Applications , 2009 .

[62]  S. Schechter.  ITERATION METHODS FOR NONLINEAR PROBLEMS , 1962 .

[63]  Tobias Jahnke,et al.  Solving chemical master equations by adaptive wavelet compression , 2010, J. Comput. Phys..

[64]  H. Keller.  On the Solution of Singular and Semidefinite Linear Systems by Iteration , 1965 .

[65]  Pierre Comon,et al.  Enhanced Line Search: A Novel Method to Accelerate PARAFAC , 2008, SIAM J. Matrix Anal. Appl..

[66]  K. Ritter,et al.  Variable Subspace Sampling and Multi-level Algorithms , 2009 .

[67]  R. Schneider,et al.  The Alternating Linear Scheme for Tensor Optimisation in the TT Format , 2022 .

[68]  Christian Bender,et al.  Dual pricing of multi-exercise options under volume constraints , 2011, Finance Stochastics.

[69]  K. Ritter,et al.  Adaptive Wavelet Methods for Elliptic Stochastic Partial Differential Equations , 2022 .

[70]  Winfried Sickel,et al.  Best m-term approximation and Lizorkin-Triebel spaces , 2011, J. Approx. Theory.

[71]  Richard A. Harshman,et al.  Determination and Proof of Minimum Uniqueness Conditions for PARAFAC1 , 1972 .

[72]  P. Paatero.  Construction and analysis of degenerate PARAFAC models , 2000 .

[73]  Fred J. Hickernell,et al.  Multi-level Monte Carlo algorithms for infinite-dimensional integration on RN , 2010, J. Complex..

[74]  Fred J. Hickernell,et al.  Deterministic multi-level algorithms for infinite-dimensional integration on RN , 2011, J. Complex..

[75]  H. Yserentant,et al.  On the Complexity of the Electronic Schrödinger Equation , 2022 .

[76]  Luigi Grippo,et al.  On the convergence of the block nonlinear Gauss-Seidel method under convex constraints , 2000, Oper. Res. Lett..

[77]  E. Novak,et al.  Optimal Order of Convergence and (In)Tractability of Multivariate Approximation of Smooth Functions , 2009 .

[78]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[79]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[80]  Wolfgang Dahmen,et al.  Compressed Sensing and Electron Microscopy , 2010 .

[81]  Wolfgang Hackbusch,et al.  Tensorisation of vectors and their efficient convolution , 2011, Numerische Mathematik.

[82]  Klaus Ritter,et al.  A Local Refinement Strategy for Constructive Quantization of Scalar SDEs , 2013, Found. Comput. Math..

[83]  G. Teschke,et al.  Inversion of the noisy Radon transform on SO(3) by Gabor frames and sparse recovery principles , 2011 .

[84]  Markus Hansen,et al.  On tensor products of quasi-Banach spaces , 2010 .

[85]  G. Teschke,et al.  Compressive sensing principles and iterative sparse recovery for inverse and ill-posed problems , 2010 .

[86]  Wang-Q Lim,et al.  Shearlets on Bounded Domains , 2010, 1007.3039.

[87]  E. Ullmann,et al.  Preconditioning Stochastic Galerkin Saddle Point Problems , 2022 .

[88]  Armin Iske,et al.  Optimally Sparse Image Representation by the Easy Path Wavelet Transform , 2012, Int. J. Wavelets Multiresolution Inf. Process..

[89]  G. Teschke,et al.  A compressive Landweber iteration for solving ill-posed inverse problems , 2008 .

[90]  Erwan Faou,et al.  Computing Semiclassical Quantum Dynamics with Hagedorn Wavepackets , 2009, SIAM J. Sci. Comput..

[91]  Lars Grasedyck,et al.  Polynomial Approximation in Hierarchical Tucker Format by Vector – Tensorization , 2010 .

[92]  Wolfgang Hackbusch,et al.  An Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples , 2011, Comput. Methods Appl. Math..

[93]  Harry Yserentant,et al.  The mixed regularity of electronic wave functions in fractional order and weighted Sobolev spaces , 2012, Numerische Mathematik.

[94]  E. Novak,et al.  On the power of function values for the approximation problem in various settings , 2010, 1011.3682.

[95]  Stephan Dahlke,et al.  Adaptive wavelet methods and sparsity reconstruction for inverse heat conduction problems , 2010, Adv. Comput. Math..

[96]  Gerlind Plonka-Hoch,et al.  The Easy Path Wavelet Transform: A New Adaptive Wavelet Transform for Sparse Representation of Two-Dimensional Data , 2008, Multiscale Model. Simul..

[97]  Gabriele Steidl,et al.  Shearlet Coorbit Spaces: Compactly Supported Analyzing Shearlets, Traces and Embeddings , 2011 .

[98]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[99]  Wolfgang Dahmen,et al.  Convergence Rates for Greedy Algorithms in Reduced Basis Methods , 2010, SIAM J. Math. Anal..

[100]  W. Greub.  Linear Algebra , 1981 .

[101]  M. Fornasier,et al.  Multilevel Preconditioning for Adaptive Sparse Optimization , 2009 .

[102]  Winfried Sickel,et al.  On Besov regularity of solutions to nonlinear elliptic partial differential equations , 2020, Nonlinear Analysis.

[103]  Wolfgang Dahmen,et al.  Fast high-dimensional approximation with sparse occupancy trees , 2011, J. Comput. Appl. Math..

[104]  Christian Bender,et al.  Least-Squares Monte Carlo for Backward SDEs , 2012 .

[105]  Daniela Rosca,et al.  A New Hybrid Method for Image Approximation Using the Easy Path Wavelet Transform , 2011, IEEE Transactions on Image Processing.

[106]  Tamara G. Kolda,et al.  Orthogonal Tensor Decompositions , 2000, SIAM J. Matrix Anal. Appl..

[107]  Gerlind Plonka-Hoch,et al.  Curvelet-Wavelet Regularized Split Bregman Iteration for Compressed Sensing , 2011, Int. J. Wavelets Multiresolution Inf. Process..

[108]  Andreas Zeiser,et al.  Wavelet Approximation in Weighted Sobolev Spaces of Mixed Order with Applications to the Electronic Schrödinger Equation , 2012 .

[109]  D. Lorenz,et al.  Greedy solution of ill-posed problems: error bounds and exact inversion , 2009, 0904.0154.

[110]  Pierre Comon,et al.  Nonnegative approximations of nonnegative tensors , 2009, ArXiv.

[111]  R. Schilling,et al.  Weak Order for the Discretization of the Stochastic Heat Equation Driven by Impulsive Noise , 2009, 0911.4681.

[112]  Armin Iske,et al.  Adaptive ADER Methods Using Kernel-Based Polyharmonic Spline WENO Reconstruction , 2010, SIAM J. Sci. Comput..

[113]  P. Comon,et al.  Tensor decompositions, alternating least squares and other tales , 2009 .

[114]  H. Harbrecht,et al.  On the low-rank approximation by the pivoted Cholesky decomposition , 2012 .

[115]  Martin J. Mohlenkamp.  Musings on multilinear fitting , 2013 .

[116]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[117]  Martin J. Mohlenkamp,et al.  Trigonometric identities and sums of separable functions , 2005 .

[118]  Karsten Urban,et al.  Adaptive Wavelet Methods on Unbounded Domains , 2012, Journal of Scientific Computing.

[119]  Luigi Grippof,et al.  Globally convergent block-coordinate techniques for unconstrained optimization , 1999 .