Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables

Neural processes (NPs) constitute a family of variational approximate models for stochastic processes with promising properties in computational efficiency and uncertainty quantification. These processes use neural networks with latent variable inputs to induce predictive distributions. However, the expressiveness of vanilla NPs is limited as they only use a global latent variable, while target specific local variation may be crucial sometimes. To address this challenge, we investigate NPs systematically and present a new variant of NP model that we call Doubly Stochastic Variational Neural Process (DSVNP). This model combines the global latent variable and local latent variables for prediction. We evaluate this model in several experiments, and our results demonstrate competitive prediction performance in multi-output regression and uncertainty estimation in classification.

[1]  Andrew Gordon Wilson,et al.  Stochastic Variational Deep Kernel Learning , 2016, NIPS.

[2]  Tim G. J. Rudner.  On the Connection between Neural Processes and Gaussian Processes with Deep Kernels , 2018 .

[3]  Neil D. Lawrence,et al.  Bayesian Gaussian Process Latent Variable Model , 2010, AISTATS.

[4]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[5]  Antonio Artés-Rodríguez,et al.  Heterogeneous Multi-output Gaussian Process Prediction , 2018, NeurIPS.

[6]  Eduard Hofer,et al.  An approximate epistemic uncertainty analysis approach in the presence of epistemic and aleatory uncertainties , 2002, Reliab. Eng. Syst. Saf..

[7]  Marc Peter Deisenroth,et al.  Doubly Stochastic Variational Inference for Deep Gaussian Processes , 2017, NIPS.

[8]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[9]  Dustin Tran,et al.  Hierarchical Implicit Models and Likelihood-Free Variational Inference , 2017, NIPS.

[10]  Koray Kavukcuoglu,et al.  Neural scene representation and rendering , 2018, Science.

[11]  Carl E. Rasmussen,et al.  Warped Gaussian Processes , 2003, NIPS.

[12]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[13]  Guodong Zhang,et al.  Functional Variational Bayesian Neural Networks , 2019, ICLR.

[14]  Dustin Tran,et al.  Variational Gaussian Process , 2015, ICLR.

[15]  Benjamin Van Roy,et al.  Deep Exploration via Bootstrapped DQN , 2016, NIPS.

[16]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[17]  Sanjay Thakur,et al.  Uncertainty Aware Learning from Demonstrations in Multiple Contexts using Bayesian Neural Networks , 2019, ICRA.

[18]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[19]  Grigorios Tsoumakas,et al.  Multi-target regression via input space expansion: treating targets as inputs , 2012, Machine Learning.

[20]  Carl E. Rasmussen,et al.  Rates of Convergence for Sparse Variational Gaussian Process Regression , 2019, ICML.

[21]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[22]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[23]  Miguel Lázaro-Gredilla,et al.  Doubly Stochastic Variational Bayes for non-Conjugate Inference , 2014, ICML.

[24]  James Hensman,et al.  Deep Gaussian Processes with Importance-Weighted Variational Inference , 2019, ICML.

[25]  Yee Whye Teh,et al.  Do Deep Generative Models Know What They Don't Know? , 2018, ICLR.

[26]  Klamer Schutte,et al.  The Functional Neural Process , 2019, NeurIPS.

[27]  Neil D. Lawrence,et al.  Variational Auto-encoded Deep Gaussian Processes , 2015, ICLR.

[28]  Edwin V. Bonilla,et al.  Scalable Inference for Gaussian Process Models with Black-Box Likelihoods , 2015, NIPS.

[29]  Sungjin Ahn,et al.  Sequential Neural Processes , 2019, NeurIPS.

[30]  I. G. MacKenzie,et al.  Stochastic Processes with Applications , 1992 .

[31]  Yee Whye Teh,et al.  Attentive Neural Processes , 2019, ICLR.

[32]  C. Rasmussen,et al.  Improving PILCO with Bayesian Neural Network Dynamics Models , 2016 .

[33]  José Miguel Hernández-Lobato,et al.  Variational Implicit Processes , 2018, ICML.

[34]  James Hensman,et al.  Scalable Variational Gaussian Process Classification , 2014, AISTATS.

[35]  Yee Whye Teh,et al.  Conditional Neural Processes , 2018, ICML.

[36]  Finale Doshi-Velez,et al.  Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning , 2017, ICML.

[37]  Andrew Gordon Wilson,et al.  Deep Kernel Learning , 2015, AISTATS.

[38]  Max Welling,et al.  Bayesian Compression for Deep Learning , 2017, NIPS.

[39]  B. Øksendal.  Stochastic differential equations : an introduction with applications , 1987 .

[40]  Shiliang Sun,et al.  Variational Dependent Multi-output Gaussian Process Dynamical Systems , 2014, J. Mach. Learn. Res..

[41]  Diederik P. Kingma,et al.  Stochastic Gradient VB and the Variational Auto-Encoder , 2013 .

[42]  Saso Dzeroski,et al.  Predicting Chemical Parameters of River Water Quality from Bioindicator Data , 2000, Applied Intelligence.

[43]  Richard E. Turner,et al.  Convolutional Conditional Neural Processes , 2020, ICLR.

[44]  Klaus H. Maier-Hein,et al.  A Probabilistic U-Net for Segmentation of Ambiguous Images , 2018, NeurIPS.

[45]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[46]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[47]  Ole Winther,et al.  Auxiliary Deep Generative Models , 2016, ICML.

[48]  Ole Winther,et al.  How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks , 2016, ICML 2016.

[49]  Carl E. Rasmussen,et al.  PILCO: A Model-Based and Data-Efficient Approach to Policy Search , 2011, ICML.

[50]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[51]  Ryan P. Adams,et al.  Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks , 2015, ICML.