Compositional uncertainty in deep Gaussian processes

Gaussian processes (GPs) are nonparametric priors over functions. Fitting a GP implies computing a posterior distribution of functions consistent with the observed data. Similarly, deep Gaussian processes (DGPs) should allow us to compute a posterior distribution of compositions of multiple functions giving rise to the observations. However, exact Bayesian inference is intractable for DGPs, motivating the use of various approximations. We show that the application of simplifying mean-field assumptions across the hierarchy leads to the layers of a DGP collapsing to near-deterministic transformations. We argue that such an inference scheme is suboptimal, not taking advantage of the potential of the model to discover the compositional structure in the data. To address this issue, we examine alternative variational inference schemes allowing for dependencies across different layers and discuss their advantages and limitations.

[1]  A. Kiureghian,et al.  Aleatory or epistemic? Does it matter? , 2009 .

[2]  Neil D. Lawrence,et al.  Hierarchical Gaussian process latent variable models , 2007, ICML '07.

[3]  Dustin Tran,et al.  TensorFlow Distributions , 2017, ArXiv.

[4]  Tim G. J. Rudner.  Inter-domain Deep Gaussian Processes , 2020, ICML.

[5]  Ryan P. Adams,et al.  Avoiding pathologies in very deep networks , 2014, AISTATS.

[6]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[7]  Andrew M. Stuart,et al.  How Deep Are Deep Gaussian Processes? , 2017, J. Mach. Learn. Res..

[8]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[9]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[10]  Marc Peter Deisenroth,et al.  Doubly Stochastic Variational Inference for Deep Gaussian Processes , 2017, NIPS.

[11]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[12]  Ieva Kazlauskaite,et al.  Gaussian Process Latent Variable Alignment Learning , 2018, AISTATS.

[13]  Andreas C. Damianou,et al.  Deep Gaussian processes and variational propagation of uncertainty , 2015 .

[14]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[15]  Neil D. Lawrence,et al.  Variational Auto-encoded Deep Gaussian Processes , 2015, ICLR.

[16]  Guodong Zhang,et al.  Functional Variational Bayesian Neural Networks , 2019, ICLR.

[17]  Neil D. Lawrence,et al.  Nested Variational Compression in Deep Gaussian Processes , 2014, 1412.1370.

[18]  C. Rasmussen,et al.  Gaussian Process Priors with Uncertain Inputs - Application to Multiple-Step Ahead Time Series Forecasting , 2002, NIPS.

[19]  Juan José Murillo-Fuentes,et al.  Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo , 2018, NeurIPS.

[20]  Ieva Kazlauskaite,et al.  Monotonic Gaussian Process Flow , 2019, ArXiv.

[21]  Hongseok Yang,et al.  On Nesting Monte Carlo Estimators , 2017, ICML.

[22]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[23]  Kurt Cutajar,et al.  Broadening the scope of gaussian processes for large-scale learning. (Élargir la portée des processus gaussiens pour l'apprentissage à grande échelle) , 2019 .

[24]  Neil D. Lawrence,et al.  Gaussian Process Latent Variable Models for Visualisation of High Dimensional Data , 2003, NIPS.

[25]  Carl E. Rasmussen,et al.  Gaussian Process Training with Input Noise , 2011, NIPS.

[26]  Miguel Lázaro-Gredilla,et al.  Bayesian Warped Gaussian Processes , 2012, NIPS.

[27]  Neil D. Lawrence,et al.  Bayesian Gaussian Process Latent Variable Model , 2010, AISTATS.

[28]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[29]  Pieter Abbeel,et al.  Inverse Reinforcement Learning via Deep Gaussian Process , 2015, UAI.

[30]  Daniel Hernández-Lobato,et al.  Deep Gaussian Processes for Regression using Approximate Expectation Propagation , 2016, ICML.

[31]  Arno Solin,et al.  Variational Fourier Features for Gaussian Processes , 2016, J. Mach. Learn. Res..

[32]  H. J. Bijl,et al.  LQG and Gaussian process techniques: For fixed-structure wind turbine control , 2018 .

[33]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[34]  Jasper Snoek,et al.  Input Warping for Bayesian Optimization of Non-Stationary Functions , 2014, ICML.

[35]  Thomas A. Runkler,et al.  Bayesian Alignments of Warped Multi-Output Gaussian Processes , 2018, NeurIPS.