Posterior inference for sparse hierarchical non-stationary models

Gaussian processes are valuable tools for non-parametric modelling, where typically an assumption of stationarity is employed. While removing this assumption can improve prediction, fitting such models is challenging. In this work, hierarchical models are constructed based on Gaussian Markov random fields with stochastic spatially varying parameters. Importantly, this allows for non-stationarity while also addressing the computational burden through a sparse banded representation of the precision matrix. In this setting, efficient Markov chain Monte Carlo (MCMC) sampling is challenging due to the strong coupling a posteriori of the parameters and hyperparameters. We develop and compare three adaptive MCMC schemes and make use of banded matrix operations for faster inference. Furthermore, a novel extension to multi-dimensional settings is proposed through an additive structure that retains the flexibility and scalability of the model, while also inheriting interpretability from the additive approach. A thorough assessment of the efficiency and accuracy of the methods in nonstationary settings is presented for both simulated experiments and a computer emulation problem.

[1]  M. Stein,et al.  Estimating deformations of isotropic Gaussian random fields on the plane , 2008, 0804.0723.

[2]  Adrian E. Raftery,et al.  [Practical Markov Chain Monte Carlo]: Comment: One Long Run with Diagnostics: Implementation Strategies for Markov Chain Monte Carlo , 1992 .

[3]  H. Rue,et al.  An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach , 2011 .

[4]  Peter Guttorp,et al.  Advances in Modeling and Inference for Environmental Processes with Nonstationary Spatial Covariance , 2001 .

[5]  Lassi Roininen,et al.  Whittle-Matérn priors for Bayesian statistical inversion with applications in electrical impedance tomography , 2014 .

[6]  Ryan P. Adams,et al.  Elliptical slice sampling , 2009, AISTATS.

[7]  Surya T. Tokdar,et al.  Computer Emulation with Nonstationary Gaussian Processes , 2016, SIAM/ASA J. Uncertain. Quantification.

[8]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[9]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[10]  Samuel Kaski,et al.  Deep learning with differential Gaussian process flows , 2018, AISTATS.

[11]  Matthias Katzfuss,et al.  Bayesian nonstationary spatial modeling for very large datasets , 2012, 1204.2098.

[12]  I. Johnstone,et al.  Adapting to Unknown Smoothness via Wavelet Shrinkage , 1995 .

[13]  Omiros Papaspiliopoulos,et al.  Auxiliary gradient‐based sampling algorithms , 2016, 1610.09641.

[14]  R. Tibshirani,et al.  Linear Smoothers and Additive Models , 1989 .

[15]  Elad Gilboa,et al.  Scaling Multidimensional Inference for Structured Gaussian Processes , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Mark D. Risser,et al.  Review: Nonstationary Spatial Modeling, with Emphasis on Process Convolution and Covariate-Driven Approaches , 2016, 1610.02447.

[17]  Faming Liang,et al.  Statistical and Computational Inverse Problems , 2006, Technometrics.

[18]  Yaming Yu,et al.  To Center or Not to Center: That Is Not the Question—An Ancillarity–Sufficiency Interweaving Strategy (ASIS) for Boosting MCMC Efficiency , 2011 .

[19]  Maurizio Filippone,et al.  A comparative evaluation of stochastic-based inference methods for Gaussian process models , 2013, Machine Learning.

[20]  Andrew M. Stuart,et al.  Dimension-Robust MCMC in Bayesian Inverse Problems , 2018 .

[21]  Roger Woodard,et al.  Interpolation of Spatial Data: Some Theory for Kriging , 1999, Technometrics.

[22]  Douglas W. Nychka,et al.  Nonstationary modeling for multivariate spatial processes , 2012, J. Multivar. Anal..

[23]  Robert B. Gramacy,et al.  tgp: An R Package for Bayesian Nonstationary, Semiparametric Nonlinear Regression and Design by Treed Gaussian Process Models , 2007 .

[24]  Juho Rousu,et al.  Non-Stationary Gaussian Process Regression with Hamiltonian Monte Carlo , 2015, AISTATS.

[25]  Mark A. Girolami,et al.  Bat Call Identification with Gaussian Process Multinomial Probit Regression and a Dynamic Time Warping Kernel , 2014, AISTATS.

[26]  F. Lindgren,et al.  Exploring a New Class of Non-stationary Spatial Gaussian Random Fields with Varying Local Anisotropy , 2013, 1304.6949.

[27]  B. Mallick,et al.  Analyzing Nonstationary Spatial Data Using Piecewise Gaussian Processes , 2005 .

[28]  Carl E. Rasmussen,et al.  Additive Gaussian Processes , 2011, NIPS.

[29]  D. Harville.  Matrix Algebra From a Statistician's Perspective , 1998 .

[30]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[31]  Richard Fallows,et al.  KAIRA: The Kilpisjärvi Atmospheric Imaging Receiver Array—System Overview and First Results , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[33]  Andrew M. Stuart,et al.  How Deep Are Deep Gaussian Processes? , 2017, J. Mach. Learn. Res..

[34]  Aki Vehtari,et al.  Hierarchical second-order analysis of replicated spatial point patterns with non-spatial covariates , 2014 .

[35]  Theodore Kypraios,et al.  Bayesian non-parametric inference for stochastic epidemic models using Gaussian Processes , 2016, Biostatistics.

[36]  Joaquim H. Vianna Neto,et al.  Accounting for spatially varying directional effects in spatial covariance structures , 2012, 1209.5977.

[37]  Christopher J Paciorek,et al.  Spatial modelling using a new class of nonstationary covariance functions , 2006, Environmetrics.

[38]  Dorit Hammerling,et al.  A Case Study Competition Among Methods for Analyzing Large Spatial Data , 2017, Journal of Agricultural, Biological and Environmental Statistics.

[39]  Aki Vehtari,et al.  An additive Gaussian process regression model for interpretable non-parametric analysis of longitudinal data , 2019, Nature Communications.

[40]  Jacques Rivoirard,et al.  A Generalized Convolution Model and Estimation for Non-stationary Random Functions , 2014, 1412.1373.

[41]  Andrew M. Stuart,et al.  Robust MCMC Sampling with Non-Gaussian and Hierarchical Priors in High Dimensions , 2018 .

[42]  H. Rue,et al.  An explicit link between Gaussian fields and Gaussian Markov random fields; The SPDE approach , 2010 .

[43]  Gareth O. Roberts,et al.  Examples of Adaptive MCMC , 2009 .

[44]  Elad Gilboa,et al.  Scaling Multidimensional Gaussian Processes using Projected Additive Approximations , 2013, ICML.

[45]  Haavard Rue,et al.  Does non-stationary spatial data always require non-stationary random fields? , 2014 .

[46]  M. Girolami,et al.  Hyperpriors for Matérn fields with applications in Bayesian inversion , 2016, Inverse Problems & Imaging.

[47]  James Hensman,et al.  Banded Matrix Operators for Gaussian Markov Models in the Automatic Differentiation Era , 2019, AISTATS.

[48]  Wolfram Burgard,et al.  Adaptive Non-Stationary Kernel Regression for Terrain Modeling , 2007, Robotics: Science and Systems.

[49]  Daniel B. Williamson,et al.  Diagnostics-Driven Nonstationary Emulators Using Kernel Mixtures , 2018, SIAM/ASA J. Uncertain. Quantification.

[50]  Samuel Kaski,et al.  Deep convolutional Gaussian processes , 2018, ECML/PKDD.

[51]  Catherine A. Calder,et al.  Local likelihood estimation for covariance functions with spatially-varying parameters: the convoSPAT package for R , 2015, 1507.08613.

[52]  Alexis Boukouvalas,et al.  GPflow: A Gaussian Process Library using TensorFlow , 2016, J. Mach. Learn. Res..

[53]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[54]  Ryan P. Adams,et al.  Avoiding pathologies in very deep networks , 2014, AISTATS.

[55]  Haotian Hang,et al.  Inconsistent Estimation and Asymptotically Equal Interpolations in Model-Based Geostatistics , 2004 .

[56]  John Cunningham,et al.  Elliptical Slice Sampling with Expectation Propagation , 2016, UAI.

[57]  Ryan P. Adams,et al.  Slice sampling covariance hyperparameters of latent Gaussian models , 2010, NIPS.

[58]  G. Matheron.  The intrinsic random functions and their applications , 1973, Advances in Applied Probability.

[59]  Haavard Rue,et al.  Bayesian Adaptive Smoothing Spline using Stochastic Differential Equations , 2012, 1209.2013.

[60]  Robert B. Gramacy,et al.  laGP: Large-Scale Spatial Modeling via Local Approximate Gaussian Processes in R , 2016 .

[61]  M. Vannucci,et al.  Covariance structure of wavelet coefficients: theory and models in a Bayesian perspective , 1999 .

[62]  A. Raftery,et al.  Probabilistic Weather Forecasting for Winter Road Maintenance , 2010 .