Variational Bayes with synthetic likelihood

Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approach uses stochastic gradient variational inference methods for posterior approximation in the synthetic likelihood context, employing unbiased estimates of the log likelihood. We compare the new method with a related likelihood-free variational inference technique in the literature, while at the same time improving the implementation of that approach in a number of ways. These new algorithms are feasible to implement in situations which are challenging for conventional approximate Bayesian computation methods, in terms of the dimensionality of the parameter and summary statistic.

[1]  A. Doucet,et al.  Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator , 2012, 1210.1871.

[2]  David J. Nott,et al.  Gaussian variational approximation with sparse precision matrices , 2016, Statistics and Computing.

[3]  Nicolas Chopin,et al.  Expectation Propagation for Likelihood-Free Inference , 2011, 1107.5959.

[4]  Michael U. Gutmann,et al.  Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models , 2015, J. Mach. Learn. Res..

[5]  M. Wand,et al.  Explaining Variational Approximations , 2010 .

[6]  David J. Nott,et al.  Variational Bayes With Intractable Likelihood , 2015, 1503.08621.

[7]  James M. Rehg,et al.  Automatic Variational ABC , 2016, 1606.08549.

[8]  Matthew P. Wand,et al.  Fully simplified multivariate normal updates in non-conjugate variational message passing , 2014, J. Mach. Learn. Res..

[9]  Gareth W. Peters,et al.  Likelihood-free Bayesian inference for α-stable models , 2012, Comput. Stat. Data Anal..

[10]  Miguel Lázaro-Gredilla,et al.  Local Expectation Gradients for Black Box Variational Inference , 2015, NIPS.

[11]  Richard Wilkinson,et al.  Accelerating ABC methods using Gaussian processes , 2014, AISTATS.

[12]  Tim Salimans,et al.  Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression , 2012, ArXiv.

[13]  M. West,et al.  Bounded Approximations for Marginal Likelihoods , 2010 .

[14]  Robert Kohn,et al.  Fast Inference for Intractable Likelihood Problems using Variational B ayes , 2016, 1705.06679.

[15]  Anthony N. Pettitt,et al.  Likelihood-free Bayesian estimation of multivariate quantile distributions , 2011, Comput. Stat. Data Anal..

[16]  Douglas M. Bates,et al.  Unconstrained parametrizations for variance-covariance matrices , 1996, Stat. Comput..

[17]  Helen MacGillivray,et al.  Weighted quantile-based estimation for a class of transformation distributions , 2002 .

[18]  Richard G. Everitt,et al.  Bayesian model comparison with un-normalised likelihoods , 2015, Stat. Comput..

[19]  Max Welling,et al.  GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computation , 2014, UAI.

[20]  David M. Blei,et al.  Stochastic Gradient Descent as Approximate Bayesian Inference , 2017, J. Mach. Learn. Res..

[21]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[22]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[23]  Pejman Rohani,et al.  Neutrality, Cross-Immunity and Subtype Dominance in Avian Influenza Viruses , 2014, PloS one.

[24]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[25]  Dustin Tran,et al.  Automatic Differentiation Variational Inference , 2016, J. Mach. Learn. Res..

[26]  Christopher C. Drovandi,et al.  Pre-processing for approximate Bayesian computation in image analysis , 2015, Stat. Comput..

[27]  M. Gutmann,et al.  Likelihood-free inference by penalised logistic regression , 2016 .

[28]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[29]  Michael I. Jordan,et al.  Variational Bayesian Inference with Stochastic Search , 2012, ICML.

[30]  A. Pettitt,et al.  Approximate Bayesian computation using indirect inference , 2011 .

[31]  H. Joe.  Multivariate models and dependence concepts , 1998 .

[32]  Scott A. Sisson,et al.  Extending approximate Bayesian computation methods to high dimensions via a Gaussian copula model , 2015, 1504.04093.

[33]  R. Adler,et al.  A practical guide to heavy tails: statistical techniques and applications , 1998 .

[34]  Satishs Iyengar,et al.  Multivariate Models and Dependence Concepts , 1998 .

[35]  Ingram Olkin,et al.  Unbiased Estimation of Some Multivariate Probability Densities and Related Functions , 1969 .

[36]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[37]  S. Wood.  Statistical inference for noisy nonlinear ecological dynamic systems , 2010, Nature.

[38]  Florian Hartig,et al.  An Extended Empirical Saddlepoint Approximation for Intractable Likelihoods , 2016, 1601.01849.

[39]  R. A. Fisher,et al.  Statistical Tables for Biological, Agricultural and Medical Research , 1956 .

[40]  L. Tippett.  Statistical Tables: For Biological, Agricultural and Medical Research , 1954 .

[41]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[42]  R. Kohn,et al.  Regression Density Estimation With Variational Methods and Stochastic Approximation , 2012 .

[43]  Andreas Huth,et al.  Technical Note: Approximate Bayesian parameterization of a process-based tropical forest model , 2014, 1401.8205.

[44]  Matteo Fasiolo,et al.  A comparison of inferential methods for highly nonlinear state space models in ecology and epidemiology , 2014 .

[45]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[46]  Chong Wang,et al.  An Adaptive Learning Rate for Stochastic Variational Inference , 2013, ICML.

[47]  Nick C Fox,et al.  Gene-Wide Analysis Detects Two New Susceptibility Genes for Alzheimer's Disease , 2014, PLoS ONE.

[48]  J. Magnus,et al.  Matrix Differential Calculus with Applications in Statistics and Econometrics (Revised Edition) , 1999 .

[49]  Jean-Michel Marin,et al.  Approximate Bayesian computational methods , 2011, Statistics and Computing.

[50]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[51]  Miguel Lázaro-Gredilla,et al.  Doubly Stochastic Variational Bayes for non-Conjugate Inference , 2014, ICML.

[52]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[53]  David Allingham,et al.  Bayesian estimation of quantile distributions , 2009, Stat. Comput..

[54]  C. Andrieu,et al.  The pseudo-marginal approach for efficient Monte Carlo computations , 2009, 0903.5480.

[55]  M. Beaumont.  Estimation of population growth or decline in genetically monitored populations. , 2003, Genetics.

[56]  Ralph S. Silva,et al.  On Some Properties of Markov Chain Monte Carlo Simulation Methods Based on the Particle Filter , 2012 .

[57]  J. McCulloch,et al.  Simple consistent estimators of stable distribution parameters , 1986 .

[58]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[59]  David T. Frazier,et al.  Bayesian Synthetic Likelihood , 2017, 2305.05120.

[60]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[61]  S. Sisson,et al.  A comparative review of dimension reduction methods in approximate Bayesian computation , 2012, 1202.3819.

[62]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.