LOCAL CONVERGENCE OF ALTERNATING SCHEMES FOR OPTIMIZATION OF CONVEX PROBLEMS IN THE TT FORMAT

Alternating linear schemes (ALS), with the Alternating Least Squares algorithm a notable special case, provide one of the simplest and most popular choices for the treatment of optimization tasks by tensor methods. An according adaptation of ALS for the recent TT (= tensor train) format (Oseledets, 2011) has recently been investigated in (Holtz, Rohwedder, Schneider, 2011). With the present work, the positive practical experiences with TT-ALS is backed up with an according local linear convergence theory for the optimization of convex functionals J . The main assumption entering the proof is that the redundancy introduced by the TT parametrization matches the null space of the Hessian of the induced functional j = J , and we give conditions under which this assumption can be expected to hold.

[1]  Klaus Ritter,et al.  A Local Refinement Strategy for Constructive Quantization of Scalar SDEs , 2013, Found. Comput. Math..

[2]  Armin Iske,et al.  Curvature analysis of frequency modulated manifolds in dimensionality reduction , 2011 .

[3]  Wolfgang Dahmen,et al.  Compressed Sensing and Electron Microscopy , 2010 .

[4]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[5]  Markus Hansen,et al.  On tensor products of quasi-Banach spaces , 2010 .

[6]  P. Maass,et al.  An analysis of electrical impedance tomography with applications to Tikhonov regularization , 2012 .

[7]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[8]  H. Harbrecht,et al.  On the low-rank approximation by the pivoted Cholesky decomposition , 2012 .

[9]  Karsten Urban,et al.  Adaptive Wavelet Methods on Unbounded Domains , 2012, Journal of Scientific Computing.

[10]  Reinhold Schneider,et al.  Optimization problems in contracted tensor networks , 2011, Comput. Vis. Sci..

[11]  Daniel Kressner,et al.  Preconditioned Low-Rank Methods for High-Dimensional Elliptic PDE Eigenvalue Problems , 2011, Comput. Methods Appl. Math..

[12]  Gerlind Plonka-Hoch,et al.  The Easy Path Wavelet Transform: A New Adaptive Wavelet Transform for Sparse Representation of Two-Dimensional Data , 2008, Multiscale Model. Simul..

[13]  P. Maass,et al.  Function spaces and optimal currents in impedance tomography , 2011 .

[14]  G. Teschke,et al.  Accelerated projected steepest descent method for nonlinear inverse problems with sparsity constraints , 2010 .

[15]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[16]  Michael Döhler,et al.  Nonequispaced Hyperbolic Cross Fast Fourier Transform , 2010, SIAM J. Numer. Anal..

[17]  Lutz Kämmerer,et al.  Interpolation lattices for hyperbolic cross trigonometric polynomials , 2012, J. Complex..

[18]  Stephan Dahlke,et al.  An adaptive wavelet method for parameter identification problems in parabolic partial differential equations , 2022 .

[19]  Gabriele Steidl,et al.  Shearlet Coorbit Spaces: Compactly Supported Analyzing Shearlets, Traces and Embeddings , 2011 .

[20]  Armin Iske,et al.  On Groupoid C∗-Algebras, Persistent Homology and Time-Frequency Analysis , 2011 .

[21]  Daniela Rosca,et al.  A New Hybrid Method for Image Approximation Using the Easy Path Wavelet Transform , 2011, IEEE Transactions on Image Processing.

[22]  Erich Novak,et al.  The Curse of Dimensionality for Monotone and Convex Functions of Many Variables , 2010, 1011.3680.

[23]  Hermann G. Matthies,et al.  Efficient Analysis of High Dimensional Data in Tensor Formats , 2012 .

[24]  K. Ritter,et al.  Adaptive Wavelet Methods for Elliptic Stochastic Partial Differential Equations , 2022 .

[25]  Winfried Sickel,et al.  Best m-term approximation and Lizorkin-Triebel spaces , 2011, J. Approx. Theory.

[26]  Wang-Q Lim,et al.  Compactly supported shearlets are optimally sparse , 2010, J. Approx. Theory.

[27]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[28]  Erwan Faou,et al.  Computing Semiclassical Quantum Dynamics with Hagedorn Wavepackets , 2009, SIAM J. Sci. Comput..

[29]  Andreas Zeiser,et al.  Wavelet Approximation in Weighted Sobolev Spaces of Mixed Order with Applications to the Electronic Schrödinger Equation , 2012 .

[30]  Wolfgang Dahmen,et al.  Fast high-dimensional approximation with sparse occupancy trees , 2011, J. Comput. Appl. Math..

[31]  Wang-Q Lim,et al.  Shearlets on Bounded Domains , 2010, 1007.3039.

[32]  G. Teschke,et al.  Inversion of the noisy Radon transform on SO(3) by Gabor frames and sparse recovery principles , 2011 .

[33]  White,et al.  Density matrix formulation for quantum renormalization groups. , 1992, Physical review letters.

[34]  Lars Grasedyck,et al.  F ¨ Ur Mathematik in Den Naturwissenschaften Leipzig a Projection Method to Solve Linear Systems in Tensor Format a Projection Method to Solve Linear Systems in Tensor Format , 2022 .

[35]  Denis Belomestny,et al.  Multilevel dual approach for pricing American style derivatives , 2012, Finance Stochastics.

[36]  Reinhold Schneider,et al.  An analysis for the DIIS acceleration method used in quantum chemistry calculations , 2011 .

[37]  Tobias Jahnke,et al.  Solving chemical master equations by adaptive wavelet compression , 2010, J. Comput. Phys..

[38]  H. Keller.  On the Solution of Singular and Semidefinite Linear Systems by Iteration , 1965 .

[39]  Stephan Dahlke,et al.  A note on quarkonial systems and multilevel partition of unity methods , 2013 .

[40]  S. Dereich,et al.  Constructive quantization: Approximation by empirical measures , 2011, 1108.5346.

[41]  Steffen Dereich,et al.  Multilevel Monte Carlo algorithms for L\'{e}vy-driven SDEs with Gaussian correction , 2011, 1101.1369.

[42]  Erich Novak,et al.  Optimal approximation of elliptic problems by linear and nonlinear mappings IV: Errors in L2 and other norms , 2004, J. Complex..

[43]  Young-Ju,et al.  CONVERGENCE ANALYSIS ON ITERATIVE METHODS FOR SEMIDEFINITE SYSTEMS , 2008 .

[44]  O. Ernst,et al.  ON THE CONVERGENCE OF GENERALIZED POLYNOMIAL CHAOS EXPANSIONS , 2011 .

[45]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[46]  Stephan Dahlke,et al.  Piecewise tensor product wavelet bases by extensions and approximation rates , 2013, Math. Comput..

[47]  Daniel Rudolf,et al.  Error bounds for computing the expectation by Markov chain Monte Carlo , 2009, Monte Carlo Methods Appl..

[48]  R. Schilling,et al.  Weak Order for the Discretization of the Stochastic Heat Equation Driven by Impulsive Noise , 2009, 0911.4681.

[49]  NiuBen,et al.  Multi-level Monte Carlo algorithms for infinite-dimensional integration on RN , 2010 .

[50]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[51]  Armin Iske,et al.  Adaptive ADER Methods Using Kernel-Based Polyharmonic Spline WENO Reconstruction , 2010, SIAM J. Sci. Comput..

[52]  D. Lorenz,et al.  Greedy solution of ill-posed problems: error bounds and exact inversion , 2009, 0904.0154.

[53]  Wang-Q Lim,et al.  Image Separation Using Shearlets , 2011 .

[54]  Lars Grasedyck,et al.  Polynomial Approximation in Hierarchical Tucker Format by Vector – Tensorization , 2010 .

[55]  Winfried Sickel,et al.  On Besov regularity of solutions to nonlinear elliptic partial differential equations , 2020, Nonlinear Analysis.

[56]  M. Fornasier,et al.  Multilevel Preconditioning for Adaptive Sparse Optimization , 2009 .

[57]  W. Hackbusch,et al.  Black Box Low Tensor-Rank Approximation Using Fiber-Crosses , 2009 .

[58]  Fred J. Hickernell,et al.  Deterministic multi-level algorithms for infinite-dimensional integration on RN , 2011, J. Complex..

[59]  Howard C. Elman,et al.  Efficient Iterative Solvers for Stochastic Galerkin Discretizations of Log-Transformed Random Diffusion Problems , 2012, SIAM J. Sci. Comput..

[60]  Klaus Ritter,et al.  Spatial Besov Regularity for Stochastic Partial Differential Equations on Lipschitz Domains , 2010, 1011.1814.

[61]  James C. Bezdek,et al.  Convergence of Alternating Optimization , 2003, Neural Parallel Sci. Comput..

[62]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[63]  Gitta Kutyniok,et al.  Data Separation by Sparse Representations , 2011, Compressed Sensing.

[64]  Stephan Dahlke,et al.  Adaptive wavelet methods and sparsity reconstruction for inverse heat conduction problems , 2010, Adv. Comput. Math..

[65]  Lutz Kämmerer,et al.  On the stability of the hyperbolic cross discrete Fourier transform , 2011, Numerische Mathematik.

[66]  Sadegh Jokar,et al.  Sparse recovery and Kronecker products , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[67]  Armin Iske,et al.  Optimal representation of piecewise Hölder smooth bivariate functions by the Easy Path Wavelet Transform , 2013, J. Approx. Theory.

[68]  G. Teschke,et al.  A compressive Landweber iteration for solving ill-posed inverse problems , 2008 .

[69]  Winfried Sickel,et al.  Best m-Term Approximation and Sobolev–Besov Spaces of Dominating Mixed Smoothness—the Case of Compact Embeddings , 2012 .

[70]  J. Ballani,et al.  Black box approximation of tensors in hierarchical Tucker format , 2013 .

[71]  R. Schneider,et al.  The Alternating Linear Scheme for Tensor Optimisation in the TT Format , 2022 .

[72]  Vin de Silva,et al.  Tensor rank and the ill-posedness of the best low-rank approximation problem , 2006, math/0607647.

[73]  E. Novak,et al.  On the power of function values for the approximation problem in various settings , 2010, 1011.3682.

[74]  Steffen Dereich,et al.  A multilevel Monte Carlo algorithm for Lévy-driven stochastic differential equations , 2011 .

[75]  G. Kutyniok,et al.  Irregular Shearlet Frames: Geometry and Approximation Properties , 2010, 1002.2657.

[76]  Harry Yserentant,et al.  The mixed regularity of electronic wave functions multiplied by explicit correlation factors , 2011 .

[77]  Reinhold Schneider,et al.  Error estimates for the Coupled Cluster method , 2013 .

[78]  Stephan Dahlke,et al.  Spatial Besov regularity for semilinear stochastic partial differential equations on bounded Lipschitz domains , 2012, Int. J. Comput. Math..

[79]  Wang-Q Lim,et al.  Compactly Supported Shearlets , 2010, 1009.4359.

[80]  Jinchao Xu,et al.  On the Convergence of Iterative Methods for Semidefinite Linear Systems , 2006, SIAM J. Matrix Anal. Appl..

[81]  Winfried Sickel,et al.  Best m-term aproximation and tensor product of Sobolev and Besov spaces-the case of non-compact embeddings , 2010 .

[82]  Steffen Dereich,et al.  Foundations of Computational Mathematics, Budapest 2011: On the Complexity of Computing Quadrature Formulas for SDEs , 2012 .

[83]  T. Schulte-Herbrüggen,et al.  Exploiting matrix symmetries and physical symmetries in matrix product states and tensor trains , 2013, 1301.0746.

[84]  Christian Bender,et al.  Error Criteria for Numerical Solutions of Backward SDEs , 2010 .

[85]  Thorsten Rohwedder,et al.  The continuous Coupled Cluster formulation for the electronic Schrödinger equation , 2013 .

[86]  J I Cirac,et al.  Renormalization-group transformations on quantum states. , 2004, Physical review letters.

[87]  Klaus Ritter,et al.  Derandomization of the Euler scheme for scalar stochastic differential equations , 2012, J. Complex..

[88]  Antonio Falcó,et al.  Proper generalized decomposition for nonlinear convex problems in tensor Banach spaces , 2011, Numerische Mathematik.

[89]  Tobias Jahnke,et al.  On Reduced Models for the Chemical Master Equation , 2011, Multiscale Model. Simul..

[90]  Elisabeth Ullmann,et al.  Stochastic Galerkin Matrices , 2010, SIAM J. Matrix Anal. Appl..

[91]  E. Novak,et al.  Optimal Order of Convergence and (In)Tractability of Multivariate Approximation of Smooth Functions , 2009 .

[92]  Wolfgang Dahmen,et al.  Adaptivity and variational stabilization for convection-diffusion equations∗ , 2012 .

[93]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[94]  Christian Bender,et al.  Primal and Dual Pricing of Multiple Exercise Options in Continuous Time , 2011, SIAM J. Financial Math..

[95]  Antonio Falcó,et al.  On Minimal Subspaces in Tensor Representations , 2012, Found. Comput. Math..

[96]  G. Kutyniok,et al.  Construction of Compactly Supported Shearlet Frames , 2010, 1003.5481.

[97]  Wolfgang Dahmen,et al.  Convergence Rates for Greedy Algorithms in Reduced Basis Methods , 2010, SIAM J. Math. Anal..

[98]  Christian Bender,et al.  Dual pricing of multi-exercise options under volume constraints , 2011, Finance Stochastics.

[99]  Harry Yserentant,et al.  The mixed regularity of electronic wave functions in fractional order and weighted Sobolev spaces , 2012, Numerische Mathematik.

[100]  K. Bredies,et al.  Regularization with non-convex separable constraints , 2009 .

[101]  Christian Bender,et al.  Least-Squares Monte Carlo for Backward SDEs , 2012 .

[102]  André Uschmajew,et al.  Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation , 2012, SIAM J. Matrix Anal. Appl..

[103]  Gerlind Plonka-Hoch,et al.  Curvelet-Wavelet Regularized Split Bregman Iteration for Compressed Sensing , 2011, Int. J. Wavelets Multiresolution Inf. Process..

[104]  G. Teschke,et al.  Compressive sensing principles and iterative sparse recovery for inverse and ill-posed problems , 2010 .

[105]  H. Yserentant,et al.  On the Complexity of the Electronic Schrödinger Equation , 2022 .

[106]  Simen Kvaal,et al.  Multiconfigurational time-dependent Hartree method to describe particle loss due to absorbing boundary conditions , 2011, 1102.3899.

[107]  E. Ullmann,et al.  Preconditioning Stochastic Galerkin Saddle Point Problems , 2022 .

[108]  Christopher J. Hillar,et al.  Most Tensor Problems Are NP-Hard , 2009, JACM.

[109]  Reinhold Schneider,et al.  Variational calculus with sums of elementary tensors of fixed rank , 2012, Numerische Mathematik.

[110]  Dajana Conte,et al.  Mathematical Modelling and Numerical Analysis an Error Analysis of the Multi-configuration Time-dependent Hartree Method of Quantum Dynamics , 2022 .

[111]  S. Schechter.  ITERATION METHODS FOR NONLINEAR PROBLEMS , 1962 .

[112]  Fred J. Hickernell,et al.  Multi-level Monte Carlo algorithms for infinite-dimensional integration on RN , 2010, J. Complex..

[113]  Henryk Wozniakowski,et al.  Discontinuous information in the worst case and randomized settings , 2011, 1106.2945.

[114]  Konstantin Grella,et al.  Sparse tensor spherical harmonics approximation in radiative transfer , 2011, J. Comput. Phys..

[115]  Wolfgang Hackbusch,et al.  Tensorisation of vectors and their efficient convolution , 2011, Numerische Mathematik.

[116]  Wolfgang Dahmen,et al.  Adaptive Petrov-Galerkin Methods for First Order Transport Equations , 2011, SIAM J. Numer. Anal..

[117]  Wolfgang Hackbusch,et al.  An Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples , 2011, Comput. Methods Appl. Math..

[118]  Henryk Wozniakowski,et al.  Approximation of infinitely differentiable multivariate functions is intractable , 2009, J. Complex..

[119]  Jianwei Ma,et al.  A Review of Curvelets and Recent Applications , 2009 .

[120]  W. Hackbusch,et al.  A New Scheme for the Tensor Representation , 2009 .

[121]  Armin Iske,et al.  Optimally Sparse Image Representation by the Easy Path Wavelet Transform , 2012, Int. J. Wavelets Multiresolution Inf. Process..

[122]  K. Ritter,et al.  Variable Subspace Sampling and Multi-level Algorithms , 2009 .