Convergence Problems with Generative Adversarial Networks (GANs)

Generative adversarial networks (GANs) are a novel approach to generative modelling, a task whose goal it is to learn a distribution of real data points. They have often proved difficult to train: GANs are unlike many techniques in machine learning, in that they are best described as a two-player game between a discriminator and generator. This has yielded both unreliability in the training process, and a general lack of understanding as to how GANs converge, and if so, to what. The purpose of this dissertation is to provide an account of the theory of GANs suitable for the mathematician, highlighting both positive and negative results. This involves identifying the problems when training GANs, and how topological and game-theoretic perspectives of GANs have contributed to our understanding and improved our techniques in recent years.

[1]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[2]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[4]  T. O’Neil.  Geometric Measure Theory , 2002 .

[5]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[6]  Sebastian Nowozin,et al.  The Numerics of GANs , 2017, NIPS.

[7]  S. Kakutani.  Concrete Representation of Abstract (M)-Spaces (A characterization of the Space of Continuous Functions) , 1941 .

[8]  C. Villani.  Optimal Transport: Old and New , 2008 .

[9]  Tao Xu,et al.  On the Discrimination-Generalization Tradeoff in GANs , 2017, ICLR.

[10]  Ian J. Goodfellow,et al.  NIPS 2016 Tutorial: Generative Adversarial Networks , 2016, ArXiv.

[11]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[12]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[13]  I. Glicksberg.  A FURTHER GENERALIZATION OF THE KAKUTANI FIXED POINT THEOREM, WITH APPLICATION TO NASH EQUILIBRIUM POINTS , 1952 .

[14]  Karan Singh,et al.  Efficient Regret Minimization in Non-Convex Games , 2017, ICML.

[15]  Yingyu Liang,et al.  Generalization and Equilibrium in Generative Adversarial Nets (GANs) , 2017, ICML.

[16]  Shing-Tung Yau,et al.  A Geometric View of Optimal Transportation and Generative Model , 2017, Comput. Aided Geom. Des..

[17]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[18]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[19]  K Fan,et al.  Minimax Theorems. , 1953, Proceedings of the National Academy of Sciences of the United States of America.

[20]  Paul R. Milgrom,et al.  Envelope Theorems for Arbitrary Choice Sets , 2002 .

[21]  Ariel Rubinstein,et al.  A Course in Game Theory , 1995 .

[22]  Giuliano Basso,et al.  A Hitchhiker ’ s guide to Wasserstein distances , 2015 .

[23]  D. A. Edwards.  On the Kantorovich–Rubinstein theorem , 2011 .

[24]  O. Bousquet,et al.  From optimal transport to generative modeling: the VEGAN cookbook , 2017, 1705.07642.

[25]  Mario Lucic,et al.  Are GANs Created Equal? A Large-Scale Study , 2017, NeurIPS.

[26]  E. Kreyszig.  Introductory Functional Analysis With Applications , 1978 .

[27]  Sepp Hochreiter,et al.  Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields , 2017, ICLR.

[28]  Kamalika Chaudhuri,et al.  Approximation and Convergence Properties of Generative Adversarial Learning , 2017, NIPS.

[29]  Christos H. Papadimitriou,et al.  Cycles in adversarial regularized learning , 2017, SODA.

[30]  David Pfau,et al.  Unrolled Generative Adversarial Networks , 2016, ICLR.

[31]  Andrew M. Dai,et al.  Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step , 2017, ICLR.

[32]  Jerry Li,et al.  Towards Understanding the Dynamics of Generative Adversarial Networks , 2017, ArXiv.

[33]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[34]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[35]  A. Müller.  Integral Probability Metrics and Their Generating Classes of Functions , 1997, Advances in Applied Probability.