Coupled VAE: Improved Accuracy and Robustness of a Variational Autoencoder

We present a coupled Variational Auto-Encoder (VAE) method that improves the accuracy and robustness of the probabilistic inferences on represented data. The new method models the dependency between input feature vectors (images) and weighs the outliers with a higher penalty by generalizing the original loss function to the coupled entropy function, using the principles of nonlinear statistical coupling. We evaluate the performance of the coupled VAE model using the MNIST dataset. Compared with the traditional VAE algorithm, the output images generated by the coupled VAE method are clearer and less blurry. The visualization of the input images embedded in 2D latent variable space provides a deeper insight into the structure of new model with coupled loss function: the latent variable has a smaller deviation and a more compact latent space generates the output values. We analyze the histogram of the likelihoods of the input images using the generalized mean, which measures the model's accuracy as a function of the relative risk. The neutral accuracy, which is the geometric mean and is consistent with a measure of the Shannon cross-entropy, is improved. The robust accuracy, measured by the -2/3 generalized mean, is also improved.

[1]  Jonathan M Zalger.  Application of variational autoencoders for aircraft turbomachinery design , 2017 .

[2]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[3]  Din J. Wasem,et al.  Mining of Massive Datasets , 2014 .

[4]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[5]  Trevor Darrell,et al.  Adversarial Feature Learning , 2016, ICLR.

[6]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[7]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[8]  Dustin Tran,et al.  Deep Probabilistic Programming , 2017, ICLR.

[9]  D. Mcalister,et al.  XIII. The law of the geometric mean , 1879, Proceedings of the Royal Society of London.

[10]  Bhiksha Raj,et al.  Hidden Markov Model Variational Autoencoder for Acoustic Unit Discovery , 2017, INTERSPEECH.

[11]  Kenric P. Nelson,et al.  Nonlinear statistical coupling , 2010 .

[12]  Aaron C. Courville,et al.  Adversarially Learned Inference , 2016, ICLR.

[13]  Kenric P. Nelson,et al.  On the average uncertainty for systems with nonlinear coupling , 2015, 1510.06951.

[14]  Kenric P. Nelson,et al.  Reduced Perplexity , 2016, Advances in Info-Metrics.

[15]  Murray Shanahan,et al.  Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders , 2016, ArXiv.

[16]  Yang Feng,et al.  Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications , 2018, WWW.

[17]  Tingwen Huang.  Neural Information Processing : 19th International Conference, ICONIP 2012, Doha, Qatar, November 12-15, 2012, Proceedings, Part II , 2012 .

[18]  Hossein Mobahi,et al.  Learning with a Wasserstein Loss , 2015, NIPS.

[19]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[20]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[21]  C. Tsallis.  Introduction to Nonextensive Statistical Mechanics: Approaching a Complex World , 2009 .

[22]  Charles A. Sutton,et al.  Autoencoding Variational Inference For Topic Models , 2017, ICLR.