Flexible Generative Adversarial Networks with Non-parametric Activation Functions

Generative adversarial networks (GANs) have become widespread models for complex density estimation tasks such as image generation or image-to-image synthesis. At the same time, training of GANs can suffer from several problems, either of stability or convergence, sometimes hindering their effective deployment. In this paper we investigate whether we can improve GAN training by endowing the neural network models with more flexible activation functions compared to the commonly used rectified linear unit (or its variants). In particular, we evaluate training a deep convolutional GAN wherein all hidden activation functions are replaced with a version of the kernel activation function (KAF), a recently proposed technique for learning non-parametric nonlinearities during the optimization process. On a thorough empirical evaluation on multiple image generation benchmarks, we show that the resulting architectures learn to generate visually pleasing images in a fraction of the number of the epochs, eventually converging to a better solution, even when we equalize (or even lower) the number of free parameters. Overall, this points to the importance of investigating better and more flexible architectures in the context of GANs.