Likelihood-free approximate Gibbs sampling

Likelihood-free methods such as approximate Bayesian computation (ABC) have extended the reach of statistical inference to problems with computationally intractable likelihoods. Such approaches perform well for small-to-moderate dimensional problems, but suffer a curse of dimensionality in the number of model parameters. We introduce a likelihood-free approximate Gibbs sampler that naturally circumvents the dimensionality issue by focusing on lower-dimensional conditional distributions. These distributions are estimated by flexible regression models either before the sampler is run, or adaptively during sampler implementation. As a result, and in comparison to Metropolis-Hastings-based approaches, we are able to fit substantially more challenging statistical models than would otherwise be possible. We demonstrate the sampler’s performance via two simulated examples, and a real analysis of Airbnb rental prices using a intractable high-dimensional multivariate nonlinear state-space model with a 36-dimensional latent state observed on 365 time points, which presents a real challenge to standard ABC techniques.

[1]  M. Blum Approximate Bayesian Computation: A Nonparametric Perspective , 2009, 0904.0635.

[2]  M. Beaumont,et al.  Likelihood-Free Inference of Population Structure and Local Adaptation in a Bayesian Hierarchical Model , 2010, Genetics.

[3]  Theodore Kypraios,et al.  Piecewise Approximate Bayesian Computation: fast inference for discretely observed Markov models using a factorised posterior distribution , 2015, Stat. Comput..

[4]  Michael A. West,et al.  Bayesian Forecasting and Dynamic Models (Springer Series in Statistics) , 2013 .

[5]  D. Balding,et al.  Approximate Bayesian computation in population genetics. , 2002, Genetics.

[6]  M. West,et al.  Bayesian forecasting and dynamic models , 1989 .

[7]  D. J. Nott,et al.  Approximate Bayesian Computation and Bayes’ Linear Analysis: Toward High-Dimensional ABC , 2011, 1112.4755.

[8]  D. Rubin,et al.  Fully conditional specification in multivariate imputation , 2006 .

[9]  Michele Haynes Flexible distributions and statistical models in ranking and selection procedures with applications , 1998 .

[10]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[11]  Edward H Ip,et al.  Behaviour of the Gibbs sampler when conditional distributions are potentially incompatible , 2015, Journal of statistical computation and simulation.

[12]  Sumeetpal S. Singh,et al.  Approximate Bayesian Computation for Smoothing , 2012, 1206.5208.

[13]  G. D. Rayner,et al.  Numerical maximum likelihood estimation for the g-and-k and generalized g-and-h distributions , 2002, Stat. Comput..

[14]  Giovanni Petris,et al.  Dynamic Linear Models with R , 2009 .

[15]  B. Martin PARAMETER ESTIMATION , 2012, Statistical Methods for Biomedical Research.

[16]  Mark M. Tanaka,et al.  Sequential Monte Carlo without likelihoods , 2007, Proceedings of the National Academy of Sciences.

[17]  Olivier François,et al.  Non-linear regression models for Approximate Bayesian Computation , 2008, Stat. Comput..

[18]  M. Goldstein Bayesian analysis of regression problems , 1976 .

[19]  D. J. Nott,et al.  Approximate Bayesian computation via regression density estimation , 2012, 1212.1479.

[20]  Laurent E. Calvet,et al.  Accurate Methods for Approximate Bayesian Computation Filtering , 2015 .

[21]  Jörg Drechsler,et al.  Does Convergence Really Matter , 2008 .

[22]  Paul Fearnhead,et al.  Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC , 2010, 1004.1112.

[23]  J. Møller Discussion on the paper by Feranhead and Prangle , 2012 .

[24]  A. Gelman Parameterization and Bayesian Modeling , 2004 .

[25]  Scott A. Sisson,et al.  Functional regression approximate Bayesian computation for Gaussian process density estimation , 2014, Comput. Stat. Data Anal..

[26]  Jean-Michel Marin,et al.  ABC random forests for Bayesian parameter inference , 2019, Bioinform..

[27]  J. Hartigan Linear Bayesian Methods , 1969 .

[28]  Daniel Wegmann,et al.  Likelihood-Free Inference in High-Dimensional Models , 2015, Genetics.

[29]  Christian P. Robert,et al.  Model Misspecification in ABC: Consequences and Diagnostics , 2017, 1708.01974.

[30]  M. West,et al.  Dynamic Generalized Linear Models and Bayesian Forecasting , 1985 .

[31]  Kerrie Mengersen,et al.  Approximating the likelihood in approximate Bayesian computation , 2018, 1803.06645.

[32]  Michael U. Gutmann,et al.  Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models , 2015, J. Mach. Learn. Res..

[33]  C. Robert,et al.  Approximate Bayesian Computation in State Space Models , 2014, 1409.8363.

[34]  Scott A. Sisson,et al.  Extending approximate Bayesian computation methods to high dimensions via a Gaussian copula model , 2015, 1504.04093.

[35]  Richard Gerlach,et al.  Estimating Quantile Families of Loss Distributions for Non-Life Insurance Modelling via L-Moments , 2016 .

[36]  Edward Meeds,et al.  Optimization Monte Carlo: Efficient and Embarrassingly Parallel Likelihood-Free Inference , 2015, Neural Information Processing Systems.

[37]  Helio S. Migon,et al.  Dynamic Bayesian beta models , 2011, Comput. Stat. Data Anal..

[38]  Mike West,et al.  Bayesian Learning from Marginal Data in Bionetwork Models , 2011, Statistical applications in genetics and molecular biology.

[39]  Richard G. Everitt,et al.  A rare event approach to high-dimensional approximate Bayesian computation , 2016, Statistics and Computing.

[40]  Cibele Queiroz da-Silva,et al.  Bayesian Dynamic Dirichlet Models , 2015, Commun. Stat. Simul. Comput..

[41]  Anthony N. Pettitt,et al.  Likelihood-free Bayesian estimation of multivariate quantile distributions , 2011, Comput. Stat. Data Anal..

[42]  Anthony N. Pettitt,et al.  Bayesian indirect inference using a parametric auxiliary model , 2015, 1505.03372.

[43]  Joseph Fourier,et al.  Approximate Bayesian Computation: a non-parametric perspective , 2013 .

[44]  S. A. Sisson,et al.  ABC Samplers , 2018, Handbook of Approximate Bayesian Computation.

[45]  Scott A. Sisson,et al.  Recalibration: A post-processing method for approximate Bayesian computation , 2017, Comput. Stat. Data Anal..

[46]  Adeline Samson,et al.  Coupling stochastic EM and approximate Bayesian computation for parameter inference in state-space models , 2015, Comput. Stat..

[47]  Gareth W. Peters,et al.  Bayesian Inference, Monte Carlo Sampling and Operational Risk. , 2006 .

[48]  Yanan Fan,et al.  Handbook of Approximate Bayesian Computation , 2018 .

[49]  S. Sisson,et al.  A comparative review of dimension reduction methods in approximate Bayesian computation , 2012, 1202.3819.

[50]  Christopher C. Drovandi,et al.  Variational Bayes with synthetic likelihood , 2016, Statistics and Computing.

[51]  Sumeetpal S. Singh,et al.  Filtering via approximate Bayesian computation , 2010, Statistics and Computing.

[52]  Gareth W. Peters,et al.  Parameter Estimation for Hidden Markov Models with Intractable Likelihoods , 2011, 1103.5399.

[53]  Giovanni Petris,et al.  Dynamic linear models , 2009 .

[54]  B. Arnold,et al.  Conditional specification of statistical models , 1999 .

[55]  Christian P. Robert,et al.  Componentwise approximate Bayesian computation via Gibbs-like steps , 2019, Biometrika.

[56]  S. Sisson,et al.  Diagnostic tools for approximate Bayesian computation using the coverage property , 2013, 1301.3166.

[57]  C. Andrieu,et al.  The pseudo-marginal approach for efficient Monte Carlo computations , 2009, 0903.5480.

[58]  Pierre Alquier,et al.  Noisy Monte Carlo: convergence of Markov chains with approximate transition kernels , 2014, Statistics and Computing.

[59]  Giovanni Petris,et al.  An R Package for Dynamic Linear Models , 2010 .

[60]  M. Beaumont Estimation of population growth or decline in genetically monitored populations. , 2003, Genetics.

[61]  Jeff Gill,et al.  What are Bayesian Methods , 2008 .

[62]  S. Wood Statistical inference for noisy nonlinear ecological dynamic systems , 2010, Nature.

[63]  Stef van Buuren,et al.  MICE: Multivariate Imputation by Chained Equations in R , 2011 .

[64]  Nicolas Chopin,et al.  Expectation Propagation for Likelihood-Free Inference , 2011, 1107.5959.

[65]  Ajay Jasra,et al.  Parameter Estimation in Hidden Markov Models With Intractable Likelihoods Using Sequential Monte Carlo , 2013, 1311.4117.

[66]  David J. Nott,et al.  Variational Bayes With Intractable Likelihood , 2015, 1503.08621.

[67]  S. A. Sisson,et al.  Overview of Approximate Bayesian Computation , 2018, 1802.09720.