Rank Bounds for Approximating Gaussian Densities in the Tensor-Train Format

Low rank tensor approximations have been employed successfully, for example, to build surrogate models that can be used to speed up large-scale inference problems in high dimensions. The success of this depends critically on the rank that is necessary to represent or approximate the underlying distribution. In this paper, we develop a-priori rank bounds for approximations in the functional Tensor-Train representation for the case of a Gaussian (normally distributed) model. We show that under suitable conditions on the precision matrix, we can represent the Gaussian density to high accuracy without suffering from an exponential growth of complexity as the dimension increases. Our results provide evidence of the suitability and limitations of low rank tensor methods in a simple but important model case. Numerical experiments confirm that the rank bounds capture the qualitative behavior of the rank structure when varying the parameters of the precision matrix and the accuracy of the approximation.

[1]  Sertac Karaman,et al.  A continuous analogue of the tensor-train decomposition , 2015, Computer Methods in Applied Mechanics and Engineering.

[2]  Markus Weimar.  Breaking the curse of dimensionality , 2015 .

[3]  Ivan Oseledets,et al.  Fast solution of multi-dimensional parabolic problems in the tensor train/quantized tensor train–format with initial application to the Fokker-Planck equation , 2012 .

[4]  Johnathan M. Bardsley,et al.  MCMC-Based Image Reconstruction with Uncertainty Quantification , 2012, SIAM J. Sci. Comput..

[5]  J. Miller.  Numerical Analysis , 1966, Nature.

[6]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[7]  B. Khoromskij,et al.  Tensor-structured Galerkin approximation of parametric and stochastic elliptic PDEs , 2010 .

[8]  M. Chiani,et al.  Improved exponential bounds and approximation for the Q-function with application to average error probability computation , 2002, Global Telecommunications Conference, 2002. GLOBECOM '02. IEEE.

[9]  Daniele Venturi,et al.  Dynamic tensor approximation of high-dimensional nonlinear PDEs , 2020, J. Comput. Phys..

[10]  Robert Scheichl,et al.  A Hybrid Alternating Least Squares-TT-Cross Algorithm for Parametric PDEs , 2017, SIAM/ASA J. Uncertain. Quantification.

[11]  C. Schwab,et al.  Deep neural network expression of posterior expectations in Bayesian PDE inversion , 2020, Inverse Problems.

[12]  Jo Eidsvik,et al.  Norges Teknisk-naturvitenskapelige Universitet Iterative Numerical Methods for Sampling from High Dimensional Gaussian Distributions Iterative Numerical Methods for Sampling from High Dimensional Gaussian Distributions , 2022 .

[13]  R. Schneider,et al.  Sampling-free Bayesian inversion with adaptive hierarchical tensor representations , 2018 .

[14]  W. Hackbusch.  Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[15]  U. Schollwoeck.  The density-matrix renormalization group , 2004, cond-mat/0409292.

[16]  Christoph Schwab,et al.  Low-rank tensor structure of linear diffusion operators in the TT and QTT formats☆ , 2013 .

[17]  B. Khoromskij,et al.  DMRG+QTT approach to computation of the ground state for the molecular Schrödinger operator , 2010 .

[18]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .

[19]  Endre Süli,et al.  Greedy Approximation of High-Dimensional Ornstein–Uhlenbeck Operators , 2011, Found. Comput. Math..

[20]  Ivan V. Oseledets,et al.  Fast Multidimensional Convolution in Low-Rank Tensor Formats via Cross Approximation , 2015, SIAM J. Sci. Comput..

[21]  Colin Fox,et al.  Approximation and sampling of multivariate probability distributions in the tensor train decomposition , 2018, Statistics and Computing.

[22]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[23]  J. Douglas Faires,et al.  Numerical Analysis , 1981 .

[24]  I. Oseledets.  Constructive Representation of Functions in Low-Rank Tensor Formats , 2012, Constructive Approximation.

[25]  Matthias Morzfeld,et al.  Localization for MCMC: sampling high-dimensional posterior distributions with local structure , 2017, J. Comput. Phys..

[26]  Jun S. Liu,et al.  Monte Carlo strategies in scientific computing , 2001 .

[27]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[28]  Reinhold Schneider,et al.  Adaptive stochastic Galerkin FEM with hierarchical tensor representations , 2015, Numerische Mathematik.

[29]  White,et al.  Density matrix formulation for quantum renormalization groups. , 1992, Physical review letters.

[30]  Sergey V. Dolgov,et al.  A Tensor Decomposition Algorithm for Large ODEs with Conservation Laws , 2014, Comput. Methods Appl. Math..

[31]  Youssef M. Marzouk,et al.  Spectral Tensor-Train Decomposition , 2014, SIAM J. Sci. Comput..

[32]  Zhinan Zhang,et al.  The rank of a random matrix , 2007, Appl. Math. Comput..

[33]  B. Khoromskij.  Tensor numerical methods for multidimensional PDES: theoretical analysis and initial applications , 2015 .

[34]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[35]  C. Fox,et al.  Accelerated Gibbs sampling of normal distributions using matrix splittings and polynomials , 2015, 1505.03512.

[36]  Lars Grasedyck,et al.  Polynomial Approximation in Hierarchical Tucker Format by Vector – Tensorization , 2010 .

[37]  Jaromír Šimša,et al.  The bestL2-approximation by finite sums of functions with separable variables , 1992 .

[38]  H. Rue.  Fast sampling of Gaussian Markov random fields , 2000 .

[39]  Hermann G. Matthies,et al.  Polynomial Chaos Expansion of Random Coefficients and the Solution of Stochastic Partial Differential Equations in the Tensor Train Format , 2015, SIAM/ASA J. Uncertain. Quantification.

[40]  Colin Fox,et al.  Fast Sampling in a Linear-Gaussian Inverse Problem , 2015, SIAM/ASA J. Uncertain. Quantification.

[41]  Boris N. Khoromskij,et al.  Two-Level QTT-Tucker Format for Optimized Tensor Calculus , 2013, SIAM J. Matrix Anal. Appl..

[42]  Tiangang Cui,et al.  Deep Composition of Tensor Trains using Squared Inverse Rosenblatt Transports , 2020, ArXiv.

[43]  Miroslav Fiedler,et al.  Structure ranks of matrices , 1993 .

[44]  Catherine E. Powell,et al.  An Introduction to Computational Stochastic PDEs , 2014 .

[45]  Eugene E. Tyrtyshnikov,et al.  Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions , 2009, SIAM J. Sci. Comput..

[46]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .