Adaptive Gaussian Process Approximation for Bayesian Inference with Expensive Likelihood Functions

We consider Bayesian inference problems with computationally intensive likelihood functions. We propose a Gaussian process (GP)–based method to approximate the joint distribution of the unknown parameters and the data, built on recent work (Kandasamy, Schneider, & Póczos, 2015). In particular, we write the joint density approximately as a product of an approximate posterior density and an exponentiated GP surrogate. We then provide an adaptive algorithm to construct such an approximation, where an active learning method is used to choose the design points. With numerical examples, we illustrate that the proposed method has competitive performance against existing approaches for Bayesian computation.

[1]  Marc Kennedy,et al.  Bayesian quadrature with non-normal approximating functions , 1998, Stat. Comput..

[2]  Heikki Haario,et al.  DRAM: Efficient adaptive MCMC , 2006, Stat. Comput..

[3]  Habib N. Najm,et al.  Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems , 2008, J. Comput. Phys..

[4]  A. O'Hagan,et al.  Bayesian inference for the uncertainty distribution of computer model outputs , 2002 .

[5]  Patrick R. Conrad,et al.  Accelerating Asymptotically Exact MCMC for Computationally Intensive Models via Local Approximations , 2014, 1402.1694.

[6]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[7]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[8]  Kirthevasan Kandasamy,et al.  Bayesian active learning for posterior estimation , 2015 .

[9]  Andreas Krause,et al.  Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies , 2008, J. Mach. Learn. Res..

[10]  A Tietäväinen,et al.  Bayesian inference of physiologically meaningful parameters from body sway measurements , 2017, Scientific Reports.

[11]  A. O'Hagan,et al.  Curve Fitting and Optimal Design for Prediction , 1978 .

[12]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[13]  Y. Marzouk,et al.  A stochastic collocation approach to Bayesian inference in inverse problems , 2009 .

[14]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[15]  Stuart Barber,et al.  The Rate of Convergence for Approximate Bayesian Computation , 2013, 1311.2038.

[16]  A. P. Dawid,et al.  Gaussian Processes to Speed up Hybrid Monte Carlo for Expensive Bayesian Integrals , 2003 .

[17]  Carl E. Rasmussen,et al.  Bayesian Monte Carlo , 2002, NIPS.

[18]  G. Crooks.  On Measures of Entropy and Information , 2015 .

[19]  Bruno Sudret,et al.  Spectral likelihood expansions for Bayesian inference , 2015, J. Comput. Phys..

[20]  J. Collins,et al.  Construction of a genetic toggle switch in Escherichia coli , 2000, Nature.

[21]  Bledar A. Konomi,et al.  Bayesian Treed Calibration: An Application to Carbon Capture With AX Sorbent , 2017 .

[22]  Nikos A. Vlassis,et al.  A Greedy EM Algorithm for Gaussian Mixture Learning , 2002, Neural Processing Letters.

[23]  Darren J Wilkinson,et al.  Bayesian parameter inference for stochastic biochemical network models using particle Markov chain Monte Carlo , 2011, Interface Focus.

[24]  Alex A. Gorodetsky,et al.  Mercer kernels and integrated variance experimental design: connections between Gaussian process regression and polynomial approximation , 2015, SIAM/ASA J. Uncertain. Quantification.

[25]  A. O'Hagan,et al.  Bayes–Hermite quadrature , 1991 .

[26]  H. Wynn,et al.  Maximum entropy sampling and optimal Bayesian experimental design , 2000 .

[27]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[28]  Claude E. Shannon,et al.  The mathematical theory of communication , 1950 .

[29]  A. O'Hagan,et al.  Polynomial Chaos : A Tutorial and Critique from a Statistician ’ s Perspective , 2013 .

[30]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[31]  Carl E. Rasmussen,et al.  Active Learning of Model Evidence Using Bayesian Quadrature , 2012, NIPS.

[32]  Jinglai Li,et al.  Adaptive Construction of Surrogates for the Bayesian Solution of Inverse Problems , 2013, SIAM J. Sci. Comput..

[33]  Habib N. Najm,et al.  Stochastic spectral methods for efficient Bayesian solution of inverse problems , 2005, J. Comput. Phys..

[34]  Yoshiyuki Asai,et al.  A Model of Postural Control in Quiet Standing: Robust Compensation of Delay-Induced Instability Using Intermittent Activation of Feedback Control , 2009, PloS one.

[35]  P. Deb.  Finite Mixture Models , 2008 .

[36]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[37]  N. Zabaras,et al.  Solution of inverse problems with limited forward solver evaluations: a Bayesian perspective , 2013 .

[38]  David J. C. MacKay,et al.  Information-Based Objective Functions for Active Data Selection , 1992, Neural Computation.