Gaussian Process Kernels for Pattern Discovery and Extrapolation

Gaussian processes are rich distributions over functions, which provide a Bayesian nonparametric approach to smoothing and interpolation. We introduce simple closed form kernels that can be used with Gaussian processes to discover patterns and enable extrapolation. These kernels are derived by modelling a spectral density - the Fourier transform of a kernel - with a Gaussian mixture. The proposed kernels support a broad class of stationary covariances, but Gaussian process inference remains simple and analytic. We demonstrate the proposed kernels by discovering patterns and performing long range extrapolation on synthetic examples, as well as atmospheric CO2 trends and airline passenger data. We also show that it is possible to reconstruct several popular standard covariances within our framework.

[1]  S. Bochner.  Lectures on Fourier Integrals. (AM-42) , 1959 .

[2]  R. V. Churchill,et al.  Lectures on Fourier Integrals , 1959 .

[3]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[4]  H. D. Miller,et al.  The Theory Of Stochastic Processes , 1977, The Mathematical Gazette.

[5]  L. Hörmander,et al.  The Analysis of Linear Partial Differential Operators I: Distribution Theory and Fourier Analysis , 1983 .

[6]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[7]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[8]  C. D. Keeling,et al.  Atmospheric CO 2 records from sites in the SIO air sampling network , 1994 .

[9]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[10]  David J. C. MacKay,et al.  BAYESIAN NON-LINEAR MODELING FOR THE PREDICTION COMPETITION , 1996 .

[11]  Geoffrey E. Hinton,et al.  Evaluation of Gaussian processes and other methods for non-linear regression , 1997 .

[12]  Stergios Stergiopoulos,et al.  Advanced Signal Processing Handbook: Theory and Implementation for Radar, Sonar, and Medical Imaging Real-Time Systems , 2000 .

[13]  Roger Woodard,et al.  Interpolation of Spatial Data: Some Theory for Kriging , 1999, Technometrics.

[14]  Michael E. Tipping.  Bayesian Inference: An Introduction to Principles and Practice in Machine Learning , 2003, Advanced Lectures on Machine Learning.

[15]  A. Yuille,et al.  Opinion TRENDS in Cognitive Sciences Vol.10 No.7 July 2006 Special Issue: Probabilistic models of cognition Vision as Bayesian inference: analysis by synthesis? , 2022 .

[16]  T. Griffiths,et al.  Probabilistic inference in human semantic memory , 2006, Trends in Cognitive Sciences.

[17]  Geoffrey E. Hinton,et al.  Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes , 2007, NIPS.

[18]  Iain Murray,et al.  Introduction to Gaussian Processes , 2008 .

[19]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[20]  Andrew Metcalfe,et al.  Time Series Data , 2009 .

[21]  Ryan P. Adams,et al.  Slice sampling covariance hyperparameters of latent Gaussian models , 2010, NIPS.

[22]  Chemin de Maupertuis.  Multiple Gaussian Process Models , 2010 .

[23]  David Ginsbourger,et al.  Additive Kernels for Gaussian Process Modeling , 2011, 1103.4023.

[24]  Charles Kemp,et al.  How to Grow a Mind: Statistics, Structure, and Abstraction , 2011, Science.

[25]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[26]  Andrew Gordon Wilson,et al.  Gaussian Process Regression Networks , 2011, ICML.

[27]  Ola ElBakry,et al.  TIME SERIES DATA , 2013 .

[28]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[29]  H. Suen,et al.  TIME SERIES ANALYSIS: INTRODUCTION , 2014 .