New error bounds for deep networks using sparse grids.

We prove a theorem concerning the approximation of multivariate functions by deep ReLU networks. We present new error estimates for which the curse of the dimensionality is lessened by establishing a connection with sparse grids.

[1]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[2]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[3]  Philipp Petersen,et al.  Optimal approximation of piecewise smooth functions using deep ReLU neural networks , 2017, Neural Networks.

[4]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[5]  Frauke Sprengel,et al.  Periodic interpolation and wavelets on sparse grids , 1998, Numerical Algorithms.

[6]  Lorenzo Rosasco,et al.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review , 2016, International Journal of Automation and Computing.

[7]  H. Bungartz,et al.  Sparse grids , 2004, Acta Numerica.

[8]  A. Grossmann,et al.  DECOMPOSITION OF HARDY FUNCTIONS INTO SQUARE INTEGRABLE WAVELETS OF CONSTANT SHAPE , 1984 .

[9]  E. N. Oberg.  The approximate solution of integral equations , 1935 .

[10]  Winfried Sickel,et al.  Spaces of functions of mixed smoothness and approximation from hyperbolic crosses , 2004, J. Approx. Theory.

[11]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[12]  Matus Telgarsky,et al.  Benefits of Depth in Neural Networks , 2016, COLT.

[13]  E. M. Wright,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[14]  Ohad Shamir,et al.  The Power of Depth for Feedforward Neural Networks , 2015, COLT.

[15]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[16]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[17]  Jie Shen,et al.  Sparse Spectral Approximations of High-Dimensional Problems Based on Hyperbolic Cross , 2010, SIAM J. Numer. Anal..

[18]  R. Srikant,et al.  Why Deep Neural Networks for Function Approximation? , 2016, ICLR.

[19]  H. N. Mhaskar,et al.  Neural Networks for Optimal Approximation of Smooth and Analytic Functions , 1996, Neural Computation.