Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold

This paper proposes a new high dimensional regression method by merging Gaussian process regression into a variational autoencoder framework. In contrast to other regression methods, the proposed method focuses on the case where output responses are on a complex high dimensional manifold, such as images. Our contributions are summarized as follows: (i) A new regression method estimating high dimensional image responses, which is not handled by existing regression algorithms, is proposed. (ii) The proposed regression method introduces a strategy to learn the latent space as well as the encoder and decoder so that the result of the regressed response in the latent space coincide with the corresponding response in the data space. (iii) The proposed regression is embedded into a generative model, and the whole procedure is developed by the variational autoencoder framework. We demonstrate the robustness and effectiveness of our method through a number of experiments on various visual data regression problems.

[1]  Lai-Wan Chan,et al.  Support Vector Machine Regression for Volatile Stock Market Prediction , 2002, IDEAL.

[2]  Jasper Snoek,et al.  Multi-Task Bayesian Optimization , 2013, NIPS.

[3]  Ruslan Salakhutdinov,et al.  Learning Deep Generative Models , 2009 .

[4]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[5]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[6]  Cristian Sminchisescu,et al.  Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Jason Weston,et al.  Curriculum learning , 2009, ICML '09.

[8]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[9]  Bo-Cheng Wei,et al.  Exponential Family Nonlinear Models , 1998 .

[10]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[11]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[12]  Zhe Gan,et al.  Variational Autoencoder for Deep Learning of Images, Labels and Captions , 2016, NIPS.

[13]  Hugo Larochelle,et al.  MADE: Masked Autoencoder for Distribution Estimation , 2015, ICML.

[14]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[16]  P. Holland,et al.  An Exponential Family of Probability Distributions for Directed Graphs , 1981 .

[17]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[18]  Neil D. Lawrence,et al.  Computationally Efficient Convolved Multiple Output Gaussian Processes , 2011, J. Mach. Learn. Res..

[19]  Yann LeCun,et al.  Learning to Linearize Under Uncertainty , 2015, NIPS.

[20]  Carl E. Rasmussen,et al.  The Infinite Gaussian Mixture Model , 1999, NIPS.

[21]  Wan-Chi Siu,et al.  Single image super-resolution using Gaussian process regression , 2011, CVPR 2011.

[22]  Neil D. Lawrence,et al.  Efficient Multioutput Gaussian Processes through Variational Inducing Kernels , 2010, AISTATS.

[23]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[24]  Bohyung Han,et al.  Learning Multi-domain Convolutional Neural Networks for Visual Tracking , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Uri Shalit,et al.  Deep Kalman Filters , 2015, ArXiv.

[26]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[29]  Kazuo Asakawa,et al.  Stock market prediction system with modular neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[30]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[31]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[32]  Honglak Lee,et al.  Attribute2Image: Conditional Image Generation from Visual Attributes , 2015, ECCV.

[33]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[34]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[35]  Y. Anzai,et al.  Pattern Recognition & Machine Learning , 2016 .

[36]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[37]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[38]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[39]  Neil D. Lawrence,et al.  Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data , 2003, NIPS.

[40]  Richard S. Zemel,et al.  Generative Moment Matching Networks , 2015, ICML.

[41]  L. Baum,et al.  Statistical Inference for Probabilistic Functions of Finite State Markov Chains , 1966 .

[42]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[43]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[44]  A. Lo,et al.  Stock Market Prices Do Not Follow Random Walks: Evidence from a Simple Specification Test , 1987 .

[45]  Neil D. Lawrence,et al.  Sparse Convolved Gaussian Processes for Multi-output Regression , 2008, NIPS.