Autoregressive Quantile Networks for Generative Modeling

We introduce autoregressive implicit quantile networks (AIQN), a fundamentally different approach to generative modeling than those commonly used, that implicitly captures the distribution using quantile regression. AIQN is able to achieve superior perceptual quality and improvements in evaluation metrics, without incurring a loss of sample diversity. The method can be applied to many existing models and architectures. In this work we extend the PixelCNN model with AIQN and demonstrate results on CIFAR-10 and ImageNet using Inception score, FID, non-cherry-picked samples, and inpainting results. We consistently observe that AIQN yields a highly stable algorithm that improves perceptual quality while maintaining a highly diverse distribution.

[1]  M. M. Siddiqui.  Distribution of quantiles in samples from a bivariate population , 1960 .

[2]  J. Tukey.  WHICH PART OF THE SAMPLE CONTAINS THE INFORMATION? , 1965, Proceedings of the National Academy of Sciences of the United States of America.

[3]  E. Parzen.  Nonparametric Statistical Data Modeling , 1979 .

[4]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[5]  Boris Polyak,et al.  Acceleration of stochastic approximation by averaging , 1992 .

[6]  M. C. Jones.  Estimating densities, quantiles, quantile densities and density quantiles , 1992 .

[7]  R. Koenker.  Confidence Intervals for Regression Quantiles , 1994 .

[8]  R. Koenker,et al.  Unit Root Quantile Autoregression Inference , 2004 .

[9]  Jan Dhaene,et al.  Risk Measures and Comonotonicity: A Review , 2006, Stochastic Models.

[10]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[11]  G. Carlier,et al.  Vector Quantile Regression , 2014, 1406.4643.

[12]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[13]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[14]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[15]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[16]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[17]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[18]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[19]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[20]  Pieter Abbeel,et al.  InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , 2016, NIPS.

[21]  Matthias Bethge,et al.  A note on the evaluation of generative models , 2015, ICLR.

[22]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[23]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[24]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[25]  M. Hallin,et al.  Multiple-Output Quantile Regression , 2016 .

[26]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[27]  O. Bousquet,et al.  From optimal transport to generative modeling: the VEGAN cookbook , 2017, 1705.07642.

[28]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[29]  Shakir Mohamed,et al.  Variational Approaches for Auto-Encoding Generative Adversarial Networks , 2017, ArXiv.

[30]  Stefano Ermon,et al.  Towards Deeper Understanding of Variational Autoencoding Models , 2017, ArXiv.

[31]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[32]  Xi Chen,et al.  PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications , 2017, ICLR.

[33]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[34]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[35]  Marc G. Bellemare,et al.  The Cramer Distance as a Solution to Biased Wasserstein Gradients , 2017, ArXiv.

[36]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[37]  Oriol Vinyals,et al.  Neural Discrete Representation Learning , 2017, NIPS.

[38]  Aaron C. Courville,et al.  Adversarially Learned Inference , 2016, ICLR.

[39]  Marco Cuturi,et al.  GAN and VAE from an Optimal Transport Point of View , 2017, 1706.01807.

[40]  Constantinos Daskalakis,et al.  Training GANs with Optimism , 2017, ICLR.

[41]  Han Zhang,et al.  Improving GANs Using Optimal Transport , 2018, ICLR.

[42]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[43]  Pieter Abbeel,et al.  PixelSNAIL: An Improved Autoregressive Generative Model , 2017, ICML.

[44]  Rémi Munos,et al.  Implicit Quantile Networks for Distributional Reinforcement Learning , 2018, ICML.

[45]  Marc G. Bellemare,et al.  Distributional Reinforcement Learning with Quantile Regression , 2017, AAAI.