Exemplar based Generation and Data Augmentation using Exemplar VAEs

This paper combines the advantages of parametric and non-parametric, exemplar based generative models using variational inference, yielding a new generative model called Exemplar VAE. This is a variant of VAE with a non-parametric Parzen window prior in the latent space. To sample from it, one first draws a random exemplar from training data, then stochastically transforms the exemplar into a latent code and a new observation. We propose Retrieval Augmented Training (RAT) that uses approximate nearest neighbor search in the latent space to speed up training based on a novel lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling. Experiments demonstrate the effectiveness of Exemplar VAEs on density estimation and representation learning. Further, generative data augmentation using Exemplar VAEs on permutation invariant MNIST and Fashion MNIST reduces classification error from 1.23 to 0.69 and 8.56 to 8.16.