Deep state-space Gaussian processes

This paper is concerned with a state-space approach to deep Gaussian process (DGP) regression. We construct the DGP by hierarchically putting transformed Gaussian process (GP) priors on the length scales and magnitudes of the next level of Gaussian processes in the hierarchy. The idea of the state-space approach is to represent the DGP as a non-linear hierarchical system of linear stochastic differential equations (SDEs), where each SDE corresponds to a conditional GP. The DGP regression problem then becomes a state estimation problem, and we can estimate the state efficiently with sequential methods by using the Markov property of the state-space DGP. The computational complexity scales linearly with respect to the number of measurements. Based on this, we formulate state-space MAP as well as Bayesian filtering and smoothing solutions to the DGP regression problem. We demonstrate the performance of the proposed models and methods on synthetic non-stationary signals and apply the state-space DGP to detection of the gravitational waves from LIGO measurements.

[1]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[2]  D. Florens-zmirou.  Approximate discrete-time schemes for statistics of diffusion processes , 1989 .

[3]  N. Gordon,et al.  Novel approach to nonlinear/non-Gaussian Bayesian state estimation , 1993 .

[4]  B. Øksendal.  Stochastic differential equations : an introduction with applications , 1987 .

[5]  Simo Särkkä,et al.  Hilbert-Space Reduced-Rank Methods For Deep Gaussian Processes , 2019, 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP).

[6]  Christopher J Paciorek,et al.  Spatial modelling using a new class of nonstationary covariance functions , 2006, Environmetrics.

[7]  Aki Vehtari,et al.  Expectation propagation for nonstationary heteroscedastic Gaussian process regression , 2014, 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP).

[8]  Arno Solin,et al.  Applied Stochastic Differential Equations , 2019 .

[9]  P. Guttorp,et al.  Nonparametric Estimation of Nonstationary Spatial Covariance Structure , 1992 .

[10]  A. Doucet,et al.  Particle Markov chain Monte Carlo methods , 2010 .

[11]  X. Mao,et al.  Stochastic Differential Equations and Applications , 1998 .

[12]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[13]  Shinsuke Koyama.  Projection smoothing for continuous and continuous-discrete stochastic dynamic systems , 2018, Signal Process..

[14]  Andrew Gordon Wilson,et al.  Learning Scalable Deep Kernels with Recurrent Structure , 2016, J. Mach. Learn. Res..

[15]  H. Rue,et al.  An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach , 2011 .

[16]  Mark Girolami,et al.  Posterior inference for sparse hierarchical non-stationary models , 2018, Comput. Stat. Data Anal..

[17]  S. Sarkka,et al.  Non-stationary multi-layered Gaussian priors for Bayesian inversion , 2020, Inverse Problems.

[18]  A. Doucet,et al.  Monte Carlo Smoothing for Nonlinear Time Series , 2004, Journal of the American Statistical Association.

[19]  Simon J. Godsill,et al.  On sequential Monte Carlo sampling methods for Bayesian filtering , 2000, Stat. Comput..

[20]  R. Khasminskii.  Stochastic Stability of Differential Equations , 1980 .

[21]  Hugh Salimbeni.  Deeply Non-Stationary Gaussian Processes , 2017 .

[22]  Juho Rousu,et al.  Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo , 2015, AISTATS.

[23]  Yarin Gal,et al.  Inter-domain Deep Gaussian Processes with RKHS Fourier Features , 2020, International Conference on Machine Learning.

[24]  Von Welch,et al.  Reproducing GW150914: The First Observation of Gravitational Waves From a Binary Black Hole Merger , 2016, Computing in Science & Engineering.

[25]  Simo Särkkä,et al.  Infinite-Dimensional Kalman Filtering Approach to Spatio-Temporal Gaussian Process Regression , 2012, AISTATS.

[26]  David Higdon,et al.  Non-Stationary Spatial Modeling , 2022, 2212.08043.

[27]  Simo Srkk,et al.  Bayesian Filtering and Smoothing , 2013 .

[28]  Bernard Hanzon,et al.  A differential geometric approach to nonlinear filtering: the projection filter , 1998, IEEE Trans. Autom. Control..

[29]  L. Rogers,et al.  Diffusions, Markov processes, and martingales , 1979 .

[30]  Andrew Gordon Wilson,et al.  Stochastic Variational Deep Kernel Learning , 2016, NIPS.

[31]  Kazufumi Ito,et al.  Gaussian filters for nonlinear filtering problems , 2000, IEEE Trans. Autom. Control..

[32]  A. Einstein,et al.  On gravitational waves , 1937 .

[33]  Alexander J. Smola,et al.  Heteroscedastic Gaussian process regression , 2005, ICML.

[34]  H. Kushner.  Approximations to optimal nonlinear filters , 1967, IEEE Transactions on Automatic Control.

[35]  Carl E. Rasmussen,et al.  Manifold Gaussian Processes for regression , 2014, 2016 International Joint Conference on Neural Networks (IJCNN).

[36]  Mark J. Schervish,et al.  Nonstationary Covariance Functions for Gaussian Process Regression , 2003, NIPS.

[37]  David Garfinkle,et al.  THE MATHEMATICS OF GRAVITATIONAL WAVES , 2017 .

[38]  Mónica F. Bugallo,et al.  A survey of Monte Carlo methods for parameter estimation , 2020, EURASIP J. Adv. Signal Process..

[39]  Ángel F. García-Fernández,et al.  Gaussian Process Classification Using Posterior Linearization , 2018, IEEE Signal Processing Letters.

[40]  M. Vannucci,et al.  Covariance structure of wavelet coefficients: theory and models in a Bayesian perspective , 1999 .

[41]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[42]  L. Ljung,et al.  Control theory : multivariable and nonlinear methods , 2000 .

[43]  W. Press,et al.  Gravitational waves. , 1980, Science.

[44]  Ryan P. Adams,et al.  Avoiding pathologies in very deep networks , 2014, AISTATS.

[45]  The Ligo Scientific Collaboration,et al.  Observation of Gravitational Waves from a Binary Black Hole Merger , 2016, 1602.03837.

[46]  Andrew Gordon Wilson,et al.  Deep Kernel Learning , 2015, AISTATS.

[47]  S. J. Koopman.  Discussion of `Particle Markov chain Monte Carlo methods – C. Andrieu, A. Doucet and R. Holenstein’ [Review of: Particle Markov chain Monte Carlo methods] , 2010 .

[48]  Simo Särkkä,et al.  Bayesian Filtering and Smoothing , 2013, Institute of Mathematical Statistics textbooks.

[49]  P. Kloeden,et al.  Numerical Solution of Stochastic Differential Equations , 1992 .

[50]  Arno Solin,et al.  Spatiotemporal Learning via Infinite-Dimensional Bayesian Filtering and Smoothing: A Look at Gaussian Process Regression Through Kalman Filtering , 2013, IEEE Signal Processing Magazine.

[51]  William J. Wilkinson,et al.  Fast Variational Learning in State-Space Gaussian Process Models , 2020, 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP).

[52]  Marc Peter Deisenroth,et al.  Doubly Stochastic Variational Inference for Deep Gaussian Processes , 2017, NIPS.

[53]  A. Friedman.  Stochastic Differential Equations and Applications , 1975 .

[54]  Jouni Hartikainen,et al.  Kalman filtering and smoothing solutions to temporal Gaussian process regression models , 2010, 2010 IEEE International Workshop on Machine Learning for Signal Processing.

[55]  G. Kitagawa.  Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models , 1996 .

[56]  Andrew M. Stuart,et al.  How Deep Are Deep Gaussian Processes? , 2017, J. Mach. Learn. Res..

[57]  Zhiguo Yang,et al.  Existence–uniqueness and continuation theorems for stochastic functional differential equations , 2008 .

[58]  Michael Koller.  An Introduction to Stochastic Integration , 2011 .

[59]  Xuerong Mao,et al.  The improved LaSalle-type theorems for stochastic functional differential equations , 2006 .

[60]  M. Girolami,et al.  Hyperpriors for Matérn fields with applications in Bayesian inversion , 2016, Inverse Problems & Imaging.

[61]  Carl E. Rasmussen,et al.  Warped Gaussian Processes , 2003, NIPS.

[62]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[63]  Mathieu Kessler.  Estimation of an Ergodic Diffusion from Discrete Observations , 1997 .

[64]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[65]  Roland Hostettler,et al.  Taylor Moment Expansion for Continuous-Discrete Gaussian Filtering , 2021, IEEE Transactions on Automatic Control.

[66]  Arno Solin,et al.  Sparse Algorithms for Markovian Gaussian Processes , 2021, AISTATS.

[67]  Miguel Lázaro-Gredilla,et al.  Variational Heteroscedastic Gaussian Process Regression , 2011, ICML.

[68]  Simo Särkkä,et al.  Gaussian filtering and smoothing for continuous-discrete dynamic systems , 2013, Signal Process..