Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ

We estimate the expressive power of certain deep neural networks (DNNs for short) on a class of countably-parametric, holomorphic maps [Formula: see text] on the parameter domain [Formula: see text]. Dimension-independent rates of best [Formula: see text]-term truncations of generalized polynomial chaos (gpc for short) approximations depend only on the summability exponent of the sequence of their gpc expansion coefficients. So-called [Formula: see text]-holomorphic maps [Formula: see text], with [Formula: see text] for some [Formula: see text], are known to allow gpc expansions with coefficient sequences in [Formula: see text]. Such maps arise for example as response surfaces of parametric PDEs, with applications in PDE uncertainty quantification (UQ) for many mathematical models in engineering and the sciences. Up to logarithmic terms, we establish the dimension independent approximation rate [Formula: see text] for these functions in terms of the total number [Formula: see text] of units and weights in the DNN. It follows that certain DNN architectures can overcome the curse of dimensionality when expressing possibly countably-parametric, real-valued maps with a certain degree of sparsity in the sequences of their gpc expansion coefficients. We also obtain rates of expressive power of DNNs for countably-parametric maps [Formula: see text], where [Formula: see text] is the Hilbert space [Formula: see text].

[1]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[2]  Albert Cohen,et al.  Shape Holomorphy of the Stationary Navier-Stokes Equations , 2018, SIAM J. Math. Anal..

[3]  Andrew R. Barron,et al.  Complexity Regularization with Application to Artificial Neural Networks , 1991 .

[4]  Claude Jeffrey Gittelson,et al.  Convergence Rates of Multilevel and Sparse Tensor Approximations for a Random Elliptic PDE , 2013, SIAM J. Numer. Anal..

[5]  A. Quarteroni,et al.  Numerical solution of parametrized Navier–Stokes equations by reduced basis methods , 2007 .

[6]  T. Poggio,et al.  Deep vs. shallow networks : An approximation theory perspective , 2016, ArXiv.

[7]  Michel Hervé,et al.  Analyticity in Infinite Dimensional Spaces , 1989 .

[8]  A. M. Stuart,et al.  Sparse deterministic approximation of Bayesian inverse problems , 2011, 1103.4522.

[9]  Albert Cohen,et al.  Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic sPDEs , 2010, Found. Comput. Math..

[10]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[11]  Wei Sun,et al.  A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis , 2018, Journal of The Royal Society Interface.

[12]  Albert Cohen,et al.  High-Dimensional Adaptive Sparse Polynomial Interpolation and Applications to Parametric PDEs , 2013, Foundations of Computational Mathematics.

[13]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[14]  Albert Cohen,et al.  Fully Discrete Approximation of Parametric and Stochastic Elliptic PDEs , 2017, SIAM J. Numer. Anal..

[15]  Yannis Sarantopoulos,et al.  Complexifications of real Banach spaces, polynomials and multilinear maps , 1999 .

[16]  C. Schwab,et al.  Electromagnetic wave scattering by random surfaces: Shape holomorphy , 2017 .

[17]  Albert Cohen,et al.  Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs , 2015 .

[18]  Albert Cohen,et al.  Approximation of high-dimensional parametric PDEs * , 2015, Acta Numerica.

[19]  R. DeVore,et al.  ANALYTIC REGULARITY AND POLYNOMIAL APPROXIMATION OF PARAMETRIC AND STOCHASTIC ELLIPTIC PDE'S , 2011 .

[20]  Josef Dick,et al.  Multilevel higher-order quasi-Monte Carlo Bayesian estimation , 2016, 1611.08324.

[21]  Albert Cohen,et al.  Sparse polynomial approximation of parametric elliptic PDEs. Part I: affine coefficients , 2015, 1509.07045.

[22]  Fanwei Kong,et al.  Machine learning–based 3‐D geometry reconstruction and modeling of aortic valve deformation using 3‐D computed tomography images , 2016, International journal for numerical methods in biomedical engineering.

[23]  Ken‐Ichi Funahashi,et al.  Approximate realization of identity mappings by three-layer neural networks , 1990 .

[24]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[25]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[26]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[27]  H. N. Mhaskar,et al.  Neural Networks for Optimal Approximation of Smooth and Analytic Functions , 1996, Neural Computation.