A Block-Sparse Tensor Train Format for Sample-Efficient High-Dimensional Polynomial Regression

Low-rank tensors are an established framework for the parametrization of multivariate polynomials. We propose to extend this framework by including the concept of block-sparsity to efficiently parametrize homogeneous, multivariate polynomials with low-rank tensors. This provides a representation of general multivariate polynomials as a sum of homogeneous, multivariate polynomials, represented by block-sparse, low-rank tensors. We show that this sum can be concisely represented by a single block-sparse, low-rank tensor. We further prove cases, where low-rank tensors are particularly well suited by showing that for banded symmetric tensors of homogeneous polynomials the block sizes in the block-sparse multivariate polynomial space can be bounded independent of the number of variables. We showcase this format by applying it to high-dimensional least squares regression problems where it demonstrates improved computational resource utilization and sample efficiency.

[1]  Travis E. Oliphant,et al.  Guide to NumPy , 2015 .

[2]  Vladimir A. Kazeev,et al.  Quantized tensor-structured finite elements for second-order elliptic PDEs in two dimensions , 2018, Numerische Mathematik.

[3]  W. Hackbusch.  On the Representation of Symmetric and Antisymmetric Tensors , 2016 .

[4]  F. Verstraete,et al.  Tensor product methods and entanglement optimization for ab initio quantum chemistry , 2014, 1412.5829.

[5]  Mathilde Chevreuil,et al.  A Least-Squares Method for Sparse Low Rank Approximation of Multivariate Functions , 2015, SIAM/ASA J. Uncertain. Quantification.

[6]  White,et al.  Density matrix formulation for quantum renormalization groups. , 1992, Physical review letters.

[7]  Lars Grasedyck,et al.  Stable ALS approximation in the TT-format for rank-adaptive tensor completion , 2019, Numerische Mathematik.

[8]  Reinhold Schneider,et al.  Convergence bounds for empirical nonlinear least-squares , 2021, ESAIM: Mathematical Modelling and Numerical Analysis.

[9]  Hans Zwart,et al.  An Introduction to Infinite-Dimensional Linear Systems Theory , 1995, Texts in Applied Mathematics.

[10]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[11]  K. Kunisch,et al.  Taylor expansions of the value function associated with a bilinear optimal control problem , 2017, Annales de l'Institut Henri Poincaré C, Analyse non linéaire.

[12]  Guifre Vidal,et al.  Tensor network decompositions in the presence of a global symmetry , 2009, 0907.2994.

[13]  Stefan Klus,et al.  Multidimensional Approximation of Nonlinear Dynamical Systems , 2018, Journal of Computational and Nonlinear Dynamics.

[14]  Reinhold Schneider,et al.  On manifolds of tensors of fixed TT-rank , 2012, Numerische Mathematik.

[15]  Daniel Kressner,et al.  Low-Rank Tensor Approximation for Chebyshev Interpolation in Parametric Option Pricing , 2019, SIAM J. Financial Math..

[16]  Reinhold Schneider,et al.  Variational Monte Carlo—bridging concepts of machine learning and high-dimensional partial differential equations , 2018, Advances in Computational Mathematics.

[17]  Michael Steinlechner,et al.  Riemannian Optimization for High-Dimensional Tensor Completion , 2016, SIAM J. Sci. Comput..

[18]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[19]  Stefan Klus,et al.  Tensor-based algorithms for image classification , 2019, Algorithms.

[20]  Alexander Sebastian Johannes Wolf Wolf.  Low rank tensor decompositions for high dimensional data approximation, recovery and prediction , 2019 .

[21]  Reinhold Schneider,et al.  Non-intrusive Tensor Reconstruction for High-Dimensional Random PDEs , 2019, Comput. Methods Appl. Math..

[22]  Reinhold Schneider,et al.  Adaptive stochastic Galerkin FEM with hierarchical tensor representations , 2015, Numerische Mathematik.

[23]  Christopher K. I. Williams,et al.  Modelling Frontal Discontinuities in Wind Fields , 2002 .

[24]  Markus Bachmayr,et al.  Stability of Low-Rank Tensor Representations and Structured Multilevel Preconditioning for Elliptic PDEs , 2018, Foundations of Computational Mathematics.

[25]  R. Schneider,et al.  Approximating the Stationary Hamilton-Jacobi-Bellman Equation by Hierarchical Tensor Products , 2019 .

[26]  Markus Bachmayr,et al.  Particle Number Conservation and Block Structures in Matrix Product States , 2021, ArXiv.

[27]  Reinhold Schneider,et al.  Optimization problems in contracted tensor networks , 2011, Comput. Vis. Sci..

[28]  Reinhold Schneider,et al.  The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format , 2012, SIAM J. Sci. Comput..

[29]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[30]  VLADIMIR A. KAZEEV,et al.  Low-Rank Explicit QTT Representation of the Laplace Operator and Its Inverse , 2012, SIAM J. Matrix Anal. Appl..

[31]  Lars Grasedyck,et al.  Tree Adaptive Approximation in the Hierarchical Tensor Format , 2014, SIAM J. Sci. Comput..

[32]  A. Cohen,et al.  Optimal weighted least-squares methods , 2016, 1608.00512.

[33]  David J. Schwab,et al.  Supervised Learning with Tensor Networks , 2016, NIPS.

[34]  Ivan V. Oseledets,et al.  Time Integration of Tensor Trains , 2014, SIAM J. Numer. Anal..

[35]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[36]  Martin Eigel,et al.  Pricing high-dimensional Bermudan options with hierarchical tensor formats , 2021, ArXiv.

[37]  Ivan V. Oseledets,et al.  DMRG Approach to Fast Linear Algebra in the TT-Format , 2011, Comput. Methods Appl. Math..

[38]  Karl Kunisch,et al.  Tensor Decomposition Methods for High-dimensional Hamilton-Jacobi-Bellman Equations , 2021, SIAM J. Sci. Comput..

[39]  Alex Goessmann.  Tensor network approaches for data-driven identification of non-linear dynamical laws , 2020 .

[40]  Xiu Yang,et al.  Enabling High-Dimensional Hierarchical Uncertainty Quantification by ANOVA and Tensor-Train Decomposition , 2014, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[41]  Ulrich Eggers,et al.  Introduction To Infinite Dimensional Linear Systems Theory , 2016 .

[42]  Lorenz Richter,et al.  Solving high-dimensional parabolic PDEs using the tensor train format , 2021, ICML.

[43]  Simone Göttlich,et al.  Microscopic and Macroscopic Traffic Flow Models including Random Accidents , 2021, Communications in Mathematical Sciences.

[44]  Christoph Schwab,et al.  Analytic regularity and nonlinear approximation of a class of parametric semilinear elliptic PDEs , 2013 .