Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds

Since the introduction of Generative Adversarial Networks (GANs) and Variational Autoencoders (VAE), the literature on generative modelling has witnessed an overwhelming resurgence. The impressive, yet elusive empirical performance of GANs has lead to the rise of many GAN-VAE hybrids, with the hopes of GAN level performance and additional benefits of VAE, such as an encoder for feature reduction, which is not offered by GANs. Recently, the Wasserstein Autoencoder (WAE) was proposed, achieving performance similar to that of GANs, yet it is still unclear whether the two are fundamentally different or can be further improved into a unified model. In this work, we study the $f$-GAN and WAE models and make two main discoveries. First, we find that the $f$-GAN and WAE objectives partake in a primal-dual relationship and are equivalent under some assumptions, which then allows us to explicate the success of WAE. Second, the equivalence result allows us to, for the first time, prove generalization bounds for Autoencoder models, which is a pertinent problem when it comes to theoretical analyses of generative models. Furthermore, we show that the WAE objective is related to other statistical quantities such as the $f$-divergence and in particular, upper bounded by the Wasserstein distance, which then allows us to tap into existing efficient (regularized) optimal transport solvers. Our findings thus present the first primal-dual relationship between GANs and Autoencoder models, comment on generalization abilities and make a step towards unifying these models.

[1]  Stefano Ermon,et al.  InfoVAE: Balancing Learning and Inference in Variational Autoencoders , 2019, AAAI.

[2]  Shun-ichi Amari,et al.  Information Geometry and Its Applications , 2016 .

[3]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[4]  Ian J. Goodfellow,et al.  NIPS 2016 Tutorial: Generative Adversarial Networks , 2016, ArXiv.

[5]  Shakir Mohamed,et al.  Learning in Implicit Generative Models , 2016, ArXiv.

[6]  F. Opitz.  Information geometry and its applications , 2012, 2012 9th European Radar Conference.

[7]  Frank Nielsen,et al.  Sinkhorn AutoEncoders , 2018, UAI.

[8]  Jitendra Malik,et al.  On the Implicit Assumptions of GANs , 2018, ArXiv.

[9]  A. Müller.  Integral Probability Metrics and Their Generating Classes of Functions , 1997, Advances in Applied Probability.

[10]  Kamalika Chaudhuri,et al.  Approximation and Convergence Properties of Generative Adversarial Learning , 2017, NIPS.

[11]  Tao Xu,et al.  On the Discrimination-Generalization Tradeoff in GANs , 2017, ICLR.

[12]  Eric P. Xing,et al.  On Unifying Deep Generative Models , 2017, ICLR.

[13]  Bernhard Schölkopf,et al.  Wasserstein Auto-Encoders , 2017, ICLR.

[14]  Dmitry P. Vetrov,et al.  Pairwise Augmented GANs with Adversarial Reconstruction Loss , 2018, ArXiv.

[15]  Sebastian Nowozin,et al.  Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks , 2017, ICML.

[16]  Alexander A. Alemi,et al.  Fixing a Broken ELBO , 2017, ICML.

[17]  Peter L. Bartlett,et al.  Rademacher and Gaussian Complexities: Risk Bounds and Structural Results , 2003, J. Mach. Learn. Res..

[18]  David Tse,et al.  A Convex Duality Framework for GANs , 2018, NeurIPS.

[19]  F. Bach,et al.  Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance , 2017, Bernoulli.

[20]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[21]  Y. Brenier.  Polar Factorization and Monotone Rearrangement of Vector-Valued Functions , 1991 .

[22]  Martin J. Wainwright,et al.  Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization , 2008, IEEE Transactions on Information Theory.

[23]  Richard Nock,et al.  f-GANs in an Information Geometric Nutshell , 2017, NIPS.

[24]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[25]  Aaron C. Courville,et al.  Adversarially Learned Inference , 2016, ICLR.

[26]  Shing-Tung Yau,et al.  A Geometric View of Optimal Transportation and Generative Model , 2017, Comput. Aided Geom. Des..

[27]  Sebastian Nowozin,et al.  f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization , 2016, NIPS.

[28]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[29]  Kamalika Chaudhuri,et al.  The Inductive Bias of Restricted f-GANs , 2018, ArXiv.

[30]  Lantao Yu,et al.  Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets , 2018, 1807.00751.

[31]  Lisa Borland,et al.  Ito-Langevin equations within generalized thermostatistics , 1998 .