Overcoming the Curse of Dimensionality in Density Estimation with Mixed Sobolev GANs

We propose a novel GAN framework for non-parametric density estimation with high-dimensional data. This framework is based on a novel density estimator, called the hyperbolic cross density estimator, which enjoys nice convergence properties in the mixed Sobolev spaces. As modifications of the usual Sobolev spaces, the mixed Sobolev spaces are more suitable for describing high-dimensional density functions. We prove that, unlike other existing approaches, the proposed GAN framework does not suffer the curse of dimensionality and can achieve the optimal convergence rate of $O_p(n^{-1/2})$, with $n$ data points in an arbitrary fixed dimension. We also study the universality of GANs in terms of the existence of ReLU networks which can approximate the density functions in the mixed Sobolev spaces up to any accuracy level.

[1]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[2]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[3]  Richard Hans Robert Hahnloser,et al.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit , 2000, Nature.

[4]  Yingyu Liang,et al.  Generalization and Equilibrium in Generative Adversarial Nets (GANs) , 2017, ICML.

[5]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[6]  Frances Y. Kuo,et al.  High-dimensional integration: The quasi-Monte Carlo way*† , 2013, Acta Numerica.

[7]  Yiming Yang,et al.  MMD GAN: Towards Deeper Understanding of Moment Matching Network , 2017, NIPS.

[8]  M. Zaheer,et al.  Nonparametric Density Estimation under Adversarial Losses , 2018, NeurIPS.

[9]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[10]  Richard S. Zemel,et al.  Generative Moment Matching Networks , 2015, ICML.

[11]  Stergios B. Fotopoulos,et al.  All of Nonparametric Statistics , 2007, Technometrics.

[12]  Vladimir N. Temlyakov,et al.  Hyperbolic Cross Approximation , 2016, 1601.03978.

[13]  Peter Hall,et al.  On the rate of convergence of orthogonal series density estimators , 1986 .

[14]  Taiji Suzuki,et al.  Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality , 2018, ICLR.

[15]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[16]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[17]  Barnabás Póczos,et al.  Nonparametric Density Estimation under Besov IPM Losses , 2019, ArXiv.

[18]  D. Dung.  B-Spline Quasi-Interpolation Sampling Representation and Sampling Recovery in Sobolev Spaces of Mixed Smoothness , 2016, 1603.01937.

[19]  Hongyuan Zha,et al.  Statistical Guarantees of Generative Adversarial Networks for Distribution Estimation , 2020, ArXiv.

[20]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[21]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[22]  H. Sebastian Seung,et al.  Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks , 2003, Neural Computation.

[23]  Jonathan Weed,et al.  Estimation of smooth densities in Wasserstein distance , 2019, COLT.

[24]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[25]  C. Villani.  Optimal Transport: Old and New , 2008 .

[26]  Jing Lei.  Convergence and concentration of empirical measures under Wasserstein distance in unbounded functional spaces , 2018, Bernoulli.

[27]  Barnabás Póczos,et al.  Minimax Distribution Estimation in Wasserstein Distance , 2018, ArXiv.

[28]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[29]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[30]  Peter L. Bartlett,et al.  Neural Network Learning - Theoretical Foundations , 1999 .

[31]  I. Johnstone,et al.  Density estimation by wavelet thresholding , 1996 .

[32]  Tengyuan Liang,et al.  How Well Can Generative Adversarial Networks (GAN) Learn Densities: A Nonparametric View , 2017, ArXiv.

[33]  F. Bach,et al.  Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance , 2017, Bernoulli.

[34]  Tengyuan Liang,et al.  On How Well Generative Adversarial Networks Learn Densities: Nonparametric and Parametric Results , 2018, ArXiv.

[35]  Zoubin Ghahramani,et al.  Training generative neural networks via Maximum Mean Discrepancy optimization , 2015, UAI.

[36]  Hyunjoong Kim,et al.  Functional Analysis I , 2017 .

[37]  Kamalika Chaudhuri,et al.  Approximation and Convergence Properties of Generative Adversarial Learning , 2017, NIPS.

[38]  Arkadi Nemirovski,et al.  Topics in Non-Parametric Statistics , 2000 .

[39]  Simon Mak,et al.  BdryGP: a new Gaussian process model for incorporating boundary information , 2019, 1908.08868.