An Introduction to Variational Autoencoders

Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models. In this work, we provide an introduction to variational autoencoders and some important extensions.

[1]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[2]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[3]  Surya Ganguli,et al.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics , 2015, ICML.

[4]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[5]  Honglak Lee,et al.  Attribute2Image: Conditional Image Generation from Visual Attributes , 2015, ECCV.

[6]  Oriol Vinyals,et al.  Bayesian Recurrent Neural Networks , 2017, ArXiv.

[7]  Stefano Ermon,et al.  Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models , 2017, AAAI.

[8]  Shakir Mohamed,et al.  Distribution Matching in Variational Inference , 2018, ArXiv.

[9]  Alexandre Lacoste,et al.  Neural Autoregressive Flows , 2018, ICML.

[10]  Jakub M. Tomczak,et al.  Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flow , 2017, 1706.02326.

[11]  Zhiting Hu,et al.  Improved Variational Autoencoders for Text Modeling using Dilated Convolutions , 2017, ICML.

[12]  Max Welling,et al.  The Variational Fair Autoencoder , 2015, ICLR.

[13]  Tom White,et al.  Sampling Generative Networks: Notes on a Few Effective Techniques , 2016, ArXiv.

[14]  David Duvenaud,et al.  Reinterpreting Importance-Weighted Autoencoders , 2017, ICLR.

[15]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[16]  Zhe Gan,et al.  Variational Autoencoder for Deep Learning of Images, Labels and Captions , 2016, NIPS.

[17]  Erhardt Barth,et al.  A Hybrid Convolutional Variational Autoencoder for Text Generation , 2017, EMNLP.

[18]  Radford M. Neal.  MCMC Using Hamiltonian Dynamics , 2011, 1206.1901.

[19]  Max Welling,et al.  Multiplicative Normalizing Flows for Variational Bayesian Neural Networks , 2017, ICML.

[20]  Ole Winther,et al.  Auxiliary Deep Generative Models , 2016, ICML.

[21]  Max Welling,et al.  Sylvester Normalizing Flows for Variational Inference , 2018, UAI.

[22]  Jack P. C. Kleijnen,et al.  Optimization and Sensitivity Analysis of Computer Simulation Models by the Score Function Method , 1996 .

[23]  Sergey Levine,et al.  MuProp: Unbiased Backpropagation for Stochastic Neural Networks , 2015, ICLR.

[24]  Ole Winther,et al.  Sequential Neural Models with Stochastic Layers , 2016, NIPS.

[25]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[26]  Pieter Abbeel,et al.  Variational Lossy Autoencoder , 2016, ICLR.

[27]  Diederik P. Kingma,et al.  Variational Recurrent Auto-Encoders , 2014, ICLR.

[28]  H. Bourlard,et al.  Auto-association by multilayer perceptrons and singular value decomposition , 1988, Biological Cybernetics.

[29]  Ole Winther,et al.  Ladder Variational Autoencoders , 2016, NIPS.

[30]  Aaron C. Courville,et al.  Adversarially Learned Inference , 2016, ICLR.

[31]  James T. Kwok,et al.  Fast Second Order Stochastic Backpropagation for Variational Inference , 2015, NIPS.

[32]  Ryan P. Adams,et al.  Composing graphical models with neural networks for structured representations and fast inference , 2016, NIPS.

[33]  Max Welling,et al.  Improving Variational Auto-Encoders using Householder Flow , 2016, ArXiv.

[34]  Yee Whye Teh,et al.  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables , 2016, ICLR.

[35]  Miguel Lázaro-Gredilla,et al.  Doubly Stochastic Variational Bayes for non-Conjugate Inference , 2014, ICML.

[36]  Richard E. Turner,et al.  Rényi Divergence Variational Inference , 2016, NIPS.

[37]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[38]  David Vázquez,et al.  PixelVAE: A Latent Variable Model for Natural Images , 2016, ICLR.

[39]  Marc'Aurelio Ranzato,et al.  Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition , 2010, ArXiv.

[40]  Arindam Banerjee,et al.  An Analysis of Logistic Models: Exponential Family Connections and Online Performance , 2007, SDM.

[41]  Aditya Deshpande,et al.  Learning Diverse Image Colorization , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Geoffrey E. Hinton,et al.  Attend, Infer, Repeat: Fast Scene Understanding with Generative Models , 2016, NIPS.

[43]  Ole Winther,et al.  How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks , 2016, ICML 2016.

[44]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Alán Aspuru-Guzik,et al.  Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules , 2016, ACS central science.

[46]  Michael I. Jordan,et al.  Variational Bayesian Inference with Stochastic Search , 2012, ICML.

[47]  Ariel D. Procaccia,et al.  Variational Dropout and the Local Reparameterization Trick , 2015, NIPS.

[48]  Maximilian Karl,et al.  Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data , 2016, ICLR.

[49]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[50]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[51]  Gustavo Deco,et al.  Higher Order Statistical Decorrelation without Information Loss , 1994, NIPS.

[52]  Maxine Eskénazi,et al.  Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders , 2017, ACL.

[53]  Thomas Brox,et al.  Learning to generate chairs with convolutional neural networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Karol Gregor,et al.  Neural Variational Inference and Learning in Belief Networks , 2014, ICML.

[55]  Julien Cornebise,et al.  Weight Uncertainty in Neural Network , 2015, ICML.

[56]  Roger B. Grosse,et al.  Isolating Sources of Disentanglement in Variational Autoencoders , 2018, NeurIPS.

[57]  Yoshua Bengio,et al.  Deep Generative Stochastic Networks Trainable by Backprop , 2013, ICML.

[58]  Christian Osendorfer,et al.  Learning Stochastic Recurrent Networks , 2014, NIPS 2014.

[59]  Geoffrey E. Hinton,et al.  The Helmholtz Machine , 1995, Neural Computation.

[60]  Noah D. Goodman,et al.  Amortized Inference in Probabilistic Reasoning , 2014, CogSci.

[61]  Yoshua Bengio,et al.  Bidirectional Helmholtz Machines , 2015, ICML.

[62]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[63]  Richard S. Zemel,et al.  Generative Moment Matching Networks , 2015, ICML.

[64]  Martin A. Riedmiller,et al.  Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images , 2015, NIPS.

[65]  Scott W. Linderman,et al.  Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms , 2016, AISTATS.

[66]  Dustin Tran,et al.  Hierarchical Variational Models , 2015, ICML.

[67]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[68]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[69]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[70]  Max Welling,et al.  Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors , 2016, ICML.

[71]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[72]  Tsao Yu,et al.  Voice conversion from non-parallel corpora using variational auto-encoder , 2016 .

[73]  Filip De Turck,et al.  VIME: Variational Information Maximizing Exploration , 2016, NIPS.

[74]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[75]  Ruslan Salakhutdinov,et al.  Generating Images from Captions with Attention , 2015, ICLR.

[76]  Max Welling,et al.  Markov Chain Monte Carlo and Variational Inference: Bridging the Gap , 2014, ICML.

[77]  Daan Wierstra,et al.  Deep AutoRegressive Networks , 2013, ICML.

[78]  Ole Winther,et al.  Autoencoding beyond pixels using a learned similarity metric , 2015, ICML.

[79]  Richard E. Turner,et al.  Black-box α-divergence minimization , 2016, ICML 2016.

[80]  Yu Tsao,et al.  Voice Conversion from Unaligned Corpora Using Variational Autoencoding Wasserstein Generative Adversarial Networks , 2017, INTERSPEECH.

[81]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[82]  Eric P. Xing,et al.  Controllable Text Generation , 2017, ArXiv.

[83]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[84]  Andriy Mnih,et al.  Variational Inference for Monte Carlo Objectives , 2016, ICML.

[85]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[86]  David Duvenaud,et al.  Sticking the Landing: An Asymptotically Zero-Variance Gradient Estimator for Variational Inference , 2017, ArXiv.

[87]  Shakir Mohamed,et al.  Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning , 2015, NIPS.

[88]  Hugo Larochelle,et al.  Efficient Learning of Deep Boltzmann Machines , 2010, AISTATS.

[89]  Dustin Tran,et al.  Deep Probabilistic Programming , 2017, ICLR.

[90]  Max Welling,et al.  Semi-supervised Learning with Deep Generative Models , 2014, NIPS.

[91]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[92]  Sam T. Roweis,et al.  EM Algorithms for PCA and SPCA , 1997, NIPS.

[93]  Stefano Ermon,et al.  Flow-GAN: Bridging implicit and prescribed learning in generative models , 2017, ArXiv.

[94]  Uri Shalit,et al.  Structured Inference Networks for Nonlinear State Space Models , 2016, AAAI.

[95]  Tim Salimans,et al.  A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features , 2016, ArXiv.

[96]  Matt J. Kusner,et al.  Grammar Variational Autoencoder , 2017, ICML.

[97]  Ying Tan,et al.  Variational Autoencoder for Semi-Supervised Text Classification , 2017, AAAI.

[98]  Dmitry P. Vetrov,et al.  Variational Dropout Sparsifies Deep Neural Networks , 2017, ICML.

[99]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[100]  Joshua B. Tenenbaum,et al.  Deep Convolutional Inverse Graphics Network , 2015, NIPS.

[101]  David Wingate,et al.  Automated Variational Inference in Probabilistic Programming , 2013, ArXiv.

[102]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[103]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[104]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[105]  Amos J. Storkey,et al.  Towards a Neural Statistician , 2016, ICLR.

[106]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[107]  Dustin Tran,et al.  Variational Gaussian Process , 2015, ICLR.

[108]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[109]  Koray Kavukcuoglu,et al.  Pixel Recurrent Neural Networks , 2016, ICML.

[110]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[111]  David Silver,et al.  Memory-based control with recurrent neural networks , 2015, ArXiv.

[112]  Andrew Brock,et al.  Neural Photo Editing with Introspective Adversarial Networks , 2016, ICLR.

[113]  Daan Wierstra,et al.  Towards Conceptual Compression , 2016, NIPS.

[114]  Joelle Pineau,et al.  A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues , 2016, AAAI.

[115]  Wojciech Zaremba,et al.  An Empirical Exploration of Recurrent Network Architectures , 2015, ICML.

[116]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[117]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[118]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[119]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[120]  Barnabás Póczos,et al.  Enabling Dark Energy Science with Deep Generative Models of Galaxy Images , 2016, AAAI.

[121]  Daan Wierstra,et al.  One-Shot Generalization in Deep Generative Models , 2016, ICML.

[122]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[123]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[124]  David M. Blei,et al.  The Generalized Reparameterization Gradient , 2016, NIPS.

[125]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[126]  Max Jaderberg,et al.  Unsupervised Learning of 3D Structure from Images , 2016, NIPS.

[127]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[128]  Paul Glasserman,et al.  Monte Carlo Methods in Financial Engineering , 2003 .

[129]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[130]  Demis Hassabis,et al.  Neural Episodic Control , 2017, ICML.

[131]  Tim Salimans,et al.  Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression , 2012, ArXiv.

[132]  Max Welling,et al.  Bayesian Compression for Deep Learning , 2017, NIPS.

[133]  Zoubin Ghahramani,et al.  A Theoretically Grounded Application of Dropout in Recurrent Neural Networks , 2015, NIPS.

[134]  Ziwei Liu,et al.  Semantic Facial Expression Editing using Autoencoded Flow , 2016, ArXiv.

[135]  Ralph Linsker,et al.  An Application of the Principle of Maximum Information Preservation to Linear Systems , 1988, NIPS.

[136]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[137]  Phil Blunsom,et al.  Neural Variational Inference for Text Processing , 2015, ICML.

[138]  Yuval Tassa,et al.  Learning Continuous Control Policies by Stochastic Value Gradients , 2015, NIPS.

[139]  Peter W. Glynn,et al.  Likelihood ratio gradient estimation for stochastic systems , 1990, CACM.

[140]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[141]  David Duvenaud,et al.  Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference , 2017, NIPS.

[142]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[143]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[144]  Dustin Tran,et al.  Automatic Differentiation Variational Inference , 2016, J. Mach. Learn. Res..