Low-rank methods for high-dimensional approximation and model order reduction

Tensor methods are among the most prominent tools for the numerical solution of high-dimensional problems where functions of multiple variables have to be approximated. These methods exploit the tensor structure of function spaces and apply to many problems in computational science which are formulated in tensor spaces, such as problems arising in stochastic calculus, uncertainty quantification or parametric analyses. Here, we present complexity reduction methods based on low-rank approximation methods. We analyze the problem of best approximation in subsets of low-rank tensors and discuss its connection with the problem of optimal model reduction in low-dimensional reduced spaces. We present different algorithms for computing approximations of a function in low-rank formats. In particular, we present constructive algorithms which are based either on a greedy construction of an approximation (with successive corrections in subsets of low-rank tensors) or on the greedy construction of tensor subspaces (for subspace-based low-rank formats). These algorithms can be applied for tensor compression, tensor completion or for the numerical solution of equations in low-rank tensor formats. A special emphasis is given to the solution of stochastic or parameter-dependent models. Different approaches are presented for the approximation of vector-valued or multivariate functions (identified with tensors), based on samples of the functions (black-box approaches) or on the models equations which are satisfied by the functions.

[1]  Bart Vandereycken,et al.  The geometry of algorithms using hierarchical tensors , 2013, Linear Algebra and its Applications.

[2]  Adrien Leygue,et al.  The Proper Generalized Decomposition for Advanced Numerical Simulations: A Primer , 2013 .

[3]  J. Ballani,et al.  Black box approximation of tensors in hierarchical Tucker format , 2013 .

[4]  Virginie Ehrlacher,et al.  Convergence of a greedy algorithm for high-dimensional convex nonlinear problems , 2010, 1004.0095.

[5]  Hermann G. Matthies,et al.  Galerkin methods for linear and nonlinear elliptic stochastic partial differential equations , 2005 .

[6]  Daniel Kressner,et al.  A literature survey of low‐rank tensor approximation techniques , 2013, 1302.7121.

[7]  A. Nouy Proper Generalized Decompositions and Separated Representations for the Numerical Solution of High Dimensional Stochastic Problems , 2010 .

[8]  A. Nouy Generalized spectral decomposition method for solving stochastic finite element equations : Invariant subspace problem and dedicated algorithms , 2008 .

[9]  Hans-Jörg Starkloff,et al.  ON THE CONVERGENCE OF THE STOCHASTIC GALERKIN METHOD FOR RANDOM ELLIPTIC PARTIAL DIFFERENTIAL EQUATIONS , 2013 .

[10]  Hermann G. Matthies,et al.  Solving stochastic systems with low-rank tensor compression , 2012 .

[11]  F. Chinesta,et al.  A Short Review in Model Order Reduction Based on Proper Generalized Decomposition , 2018 .

[12]  A. Nouy A generalized spectral decomposition technique to solve a class of linear stochastic partial differential equations , 2007 .

[13]  W. Hackbusch,et al.  A New Scheme for the Tensor Representation , 2009 .

[14]  T. Lelièvre,et al.  Greedy algorithms for high-dimensional non-symmetric linear problems , 2012, 1210.6688.

[15]  Grégory Legrain,et al.  Tensor-based methods for numerical homogenization from high-resolution images , 2013 .

[16]  A. Nouy A priori model reduction through Proper Generalized Decomposition for solving time-dependent partial differential equations , 2010 .

[17]  Albert Cohen,et al.  Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs , 2011 .

[18]  Wolfgang Dahmen,et al.  Adaptivity and variational stabilization for convection-diffusion equations∗ , 2012 .

[19]  Lars Grasedyck,et al.  Hierarchical Singular Value Decomposition of Tensors , 2010, SIAM J. Matrix Anal. Appl..

[20]  W. Hackbusch,et al.  On the Convergence of Alternating Least Squares Optimisation in Tensor Format Representations , 2015, 1506.00062.

[21]  Reinhold Schneider,et al.  Approximation rates for the hierarchical tensor format in periodic Sobolev spaces , 2014, J. Complex..

[22]  Albert Cohen,et al.  Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs , 2015 .

[23]  Ivan V. Oseledets,et al.  Solution of Linear Systems and Matrix Inversion in the TT-Format , 2012, SIAM J. Sci. Comput..

[24]  N. Nguyen,et al.  A general multipurpose interpolation procedure: the magic points , 2008 .

[25]  Christian Soize,et al.  Random field representations for stochastic elliptic boundary value problems and statistical inverse problems , 2013, European Journal of Applied Mathematics.

[26]  N. Nguyen,et al.  An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations , 2004 .

[27]  R. DeVore,et al.  Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs , 2010 .

[28]  Fabio Nobile,et al.  A Sparse Grid Stochastic Collocation Method for Partial Differential Equations with Random Input Data , 2008, SIAM J. Numer. Anal..

[29]  Mathilde Chevreuil,et al.  A Least-Squares Method for Sparse Low Rank Approximation of Multivariate Functions , 2015, SIAM/ASA J. Uncertain. Quantification.

[30]  W. Hackbusch,et al.  Black Box Low Tensor-Rank Approximation Using Fiber-Crosses , 2009 .

[31]  Antonio Falcó,et al.  Geometric structures in tensor representations , 2013 .

[32]  Stefan Volkwein,et al.  Galerkin proper orthogonal decomposition methods for parameter dependent elliptic systems , 2007 .

[33]  Anthony Nouy,et al.  Model Reduction Based on Proper Generalized Decomposition for the Stochastic Steady Incompressible Navier-Stokes Equations , 2014, SIAM J. Sci. Comput..

[34]  Rob P. Stevenson,et al.  Space-time adaptive wavelet methods for parabolic evolution problems , 2009, Math. Comput..

[35]  T. Roubíček Nonlinear partial differential equations with applications , 2005 .

[36]  Grégory Legrain,et al.  Low-Rank Approximate Inverse for Preconditioning Tensor-Structured Linear Systems , 2013, SIAM J. Sci. Comput..

[37]  P. Ladevèze Nonlinear Computational Structural Mechanics: New Approaches and Non-Incremental Methods of Calculation , 1998 .

[38]  J. Zolésio,et al.  Springer series in Computational Mathematics , 1992 .

[39]  Marie Billaud-Friess,et al.  A tensor approximation method based on ideal minimal residual formulations for the solution of high-dimensional problems ∗ , 2013, 1304.6126.

[40]  André Uschmajew,et al.  On Local Convergence of Alternating Schemes for Optimization of Convex Problems in the Tensor Train Format , 2013, SIAM J. Numer. Anal..

[41]  Antonio Falcó,et al.  On Minimal Subspaces in Tensor Representations , 2012, Found. Comput. Math..

[42]  Daniel Kressner,et al.  Low-Rank Tensor Krylov Subspace Methods for Parametrized Linear Systems , 2011, SIAM J. Matrix Anal. Appl..

[43]  E. Tyrtyshnikov,et al.  TT-cross approximation for multidimensional arrays , 2010 .

[44]  Boris N. Khoromskij,et al.  Approximate iterations for structured matrices , 2008, Numerische Mathematik.

[45]  Will Light,et al.  Approximation Theory in Tensor Product Spaces , 1985 .

[46]  Lars Grasedyck,et al.  F ¨ Ur Mathematik in Den Naturwissenschaften Leipzig a Projection Method to Solve Linear Systems in Tensor Format a Projection Method to Solve Linear Systems in Tensor Format , 2022 .

[47]  B. Khoromskij Tensors-structured Numerical Methods in Scientific Computing: Survey on Recent Advances , 2012 .

[48]  Wolfgang Dahmen,et al.  Convergence Rates for Greedy Algorithms in Reduced Basis Methods , 2010, SIAM J. Math. Anal..

[49]  Abderezak Touzene,et al.  A Tensor Sum Preconditioner for Stochastic Automata Networks , 2008, INFORMS J. Comput..

[50]  P. Ladevèze,et al.  The LATIN multiscale computational method and the Proper Generalized Decomposition , 2010 .

[51]  Boris N. Khoromskij,et al.  Tensor-Structured Preconditioners and Approximate Inverse of Elliptic Operators in ℝd , 2009 .

[52]  Fabio Nobile,et al.  Computers and Mathematics with Applications Convergence of Quasi-optimal Stochastic Galerkin Methods for a Class of Pdes with Random Coefficients , 2022 .

[53]  P. Frauenfelder,et al.  Finite elements for elliptic problems with stochastic coefficients , 2005 .

[54]  Reinhold Schneider,et al.  Tensor completion in hierarchical tensor representations , 2014, ArXiv.

[55]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[56]  Albert Cohen,et al.  Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic sPDEs , 2010, Found. Comput. Math..

[57]  A. Nouy,et al.  Model order reduction based on proper generalized decomposition for the propagation of uncertainties in structural dynamics , 2012 .

[58]  Christoph Schwab,et al.  Sparse Tensor Discretization of Elliptic sPDEs , 2009, SIAM J. Sci. Comput..

[59]  Fabio Nobile,et al.  An Anisotropic Sparse Grid Stochastic Collocation Method for Partial Differential Equations with Random Input Data , 2008, SIAM J. Numer. Anal..

[60]  A. Patera,et al.  A PRIORI CONVERGENCE OF THE GREEDY ALGORITHM FOR THE PARAMETRIZED REDUCED BASIS METHOD , 2012 .

[61]  Antonio Falcó,et al.  Proper generalized decomposition for nonlinear convex problems in tensor Banach spaces , 2011, Numerische Mathematik.

[62]  Wolfgang Hackbusch The Use of Sparse Grid Approximation for the r -Term Tensor Representation , 2012 .

[63]  Wolfgang Dahmen,et al.  Adaptive Near-Optimal Rank Tensor Approximation for High-Dimensional Operator Equations , 2013, Foundations of Computational Mathematics.

[64]  G. Iaccarino,et al.  Non-intrusive low-rank separated approximation of high-dimensional stochastic models , 2012, 1210.1532.

[65]  Martin J. Mohlenkamp,et al.  Multivariate Regression and Machine Learning with Sums of Separable Functions , 2009, SIAM J. Sci. Comput..

[66]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[67]  I. Babuska,et al.  Solving elliptic boundary value problems with uncertain coefficients by the finite element method: the stochastic formulation , 2005 .

[68]  V. N. Temlyakov,et al.  Greedy Approximation in Convex Optimization , 2012, Constructive Approximation.

[69]  Ronald DeVore,et al.  Greedy Algorithms for Reduced Bases in Banach Spaces , 2012, Constructive Approximation.

[70]  Robert Scheichl,et al.  Finite Element Error Analysis of Elliptic PDEs with Random Coefficients and Its Application to Multilevel Monte Carlo Methods , 2013, SIAM J. Numer. Anal..

[71]  Julia Charrier,et al.  Strong and Weak Error Estimates for Elliptic Partial Differential Equations with Random Coefficients , 2012, SIAM J. Numer. Anal..

[72]  A. Defant,et al.  Tensor Norms and Operator Ideals , 2011 .

[73]  Pierre Ladevèze,et al.  Multiscale Computational Strategy With Time and Space Homogenization: A Radial-Type Approximation Technique for Solving Microproblems , 2004 .

[74]  A. Debussche,et al.  Weak truncation error estimates for elliptic PDEs with lognormal coefficients , 2013 .

[75]  Boris N. Khoromskij,et al.  Tensor-Structured Galerkin Approximation of Parametric and Stochastic Elliptic PDEs , 2011, SIAM J. Sci. Comput..

[76]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[77]  Claude Jeffrey Gittelson,et al.  Sparse tensor discretizations of high-dimensional parametric and stochastic PDEs* , 2011, Acta Numerica.

[78]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[79]  Y. Maday,et al.  Results and Questions on a Nonlinear Approximation Approach for Solving High-dimensional Partial Differential Equations , 2008, 0811.0474.

[80]  Danny C. Sorensen,et al.  Nonlinear Model Reduction via Discrete Empirical Interpolation , 2010, SIAM J. Sci. Comput..

[81]  W. Hackbusch Tensor Spaces and Numerical Tensor Calculus , 2012, Springer Series in Computational Mathematics.

[82]  R. DeVore,et al.  ANALYTIC REGULARITY AND POLYNOMIAL APPROXIMATION OF PARAMETRIC AND STOCHASTIC ELLIPTIC PDE'S , 2011 .

[83]  Wolfgang Hackbusch,et al.  Geometric Structures in Tensor Representations (Final Release) , 2015, 1505.03027.

[84]  Albert Cohen,et al.  High-Dimensional Adaptive Sparse Polynomial Interpolation and Applications to Parametric PDEs , 2013, Foundations of Computational Mathematics.

[85]  Wolfgang Hackbusch,et al.  A regularized Newton method for the efficient approximation of tensors represented in the canonical tensor format , 2012, Numerische Mathematik.