Multilevel Linear Models, Gibbs Samplers and Multigrid Decompositions

We study the convergence properties of the Gibbs Sampler in the context of posterior distributions arising from Bayesian analysis of conditionally Gaussian hierarchical models. We develop a multigrid approach to derive analytic expressions for the convergence rates of the algorithm for various widely used model structures, including nested and crossed random effects. Our results apply to multilevel models with an arbitrary number of layers in the hierarchy, while most previous work was limited to the two-level nested case. The theoretical results provide explicit and easy-to-implement guidelines to optimize practical implementations of the Gibbs Sampler, such as indications on which parametrization to choose (e.g. centred and non-centred), which constraint to impose to guarantee statistical identifiability, and which parameters to monitor in the diagnostic process. Simulations suggest that the results are informative also in the context of non-Gaussian distributions and more general MCMC schemes, such as gradient-based ones.implementation of Gibbs samplers on conditionally Gaussian hierarchical models.

[1]  Persi Diaconis,et al.  Iterated Random Functions , 1999, SIAM Rev..

[2]  Gareth O. Roberts,et al.  On convergence of the EM algorithmand the Gibbs sampler , 1999, Stat. Comput..

[3]  Galin L. Jones,et al.  Sufficient burn-in for Gibbs samplers for a hierarchical random effects model , 2004, math/0406454.

[4]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[5]  Xiao-Li Meng,et al.  Seeking efficient data augmentation schemes via conditional and marginal augmentation , 1999 .

[6]  J. Rosenthal,et al.  Surprising Convergence Properties of Some Simple Gibbs Samplers under Various Scans , 2015 .

[7]  B. Rajaratnam,et al.  MCMC-Based Inference in the Era of Big Data: A Fundamental Analysis of the Convergence Complexity of High-Dimensional Chains , 2015, 1508.00947.

[8]  L. Brown,et al.  Empirical Bayes estimates for a two-way cross-classified model , 2018, The Annals of Statistics.

[9]  Limit theorems for iterated random functions by regenerative methods , 2001 .

[10]  S. Sain,et al.  Bayesian functional ANOVA modeling using Gaussian process prior distributions , 2010 .

[11]  G. Roberts,et al.  Updating Schemes, Correlation Structure, Blocking and Parameterization for the Gibbs Sampler , 1997 .

[12]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[13]  A. Gelfand,et al.  Efficient parametrisations for normal linear mixed models , 1995 .

[14]  A. Gelfand,et al.  Identifiability, Improper Priors, and Gibbs Sampling for Generalized Linear Models , 1999 .

[15]  Yaming Yu,et al.  To Center or Not to Center: That Is Not the Question—An Ancillarity–Sufficiency Interweaving Strategy (ASIS) for Boosting MCMC Efficiency , 2011 .

[16]  A. Owen,et al.  Efficient moment calculations for variance components in large unbalanced crossed random effects models , 2016, 1602.00346.

[17]  Gareth O. Roberts,et al.  A General Framework for the Parametrization of Hierarchical Models , 2007, 0708.3797.

[18]  Goodman,et al.  Multigrid Monte Carlo method. Conceptual foundations. , 1989, Physical review. D, Particles and fields.

[19]  Kshitij Khare,et al.  RATES OF CONVERGENCE OF SOME MULTIVARIATE MARKOV CHAINS WITH POLYNOMIAL EIGENFUNCTIONS , 2009, 0906.4242.

[20]  P. Diaconis,et al.  Stochastic Alternating Projections , 2010 .

[21]  Mark R. Bass,et al.  A comparison of centring parameterisations of Gaussian process-based models for Bayesian computation using MCMC , 2017, Stat. Comput..

[22]  Adrian F. M. Smith,et al.  Bayesian computation via the gibbs sampler and related markov chain monte carlo methods (with discus , 1993 .

[23]  Gareth O. Roberts,et al.  Complexity bounds for Markov chain Monte Carlo algorithms via diffusion limits , 2016, Journal of Applied Probability.

[24]  Y. Amit.  Convergence properties of the Gibbs sampler for perturbations of Gaussians , 1996 .

[25]  Gareth O. Roberts,et al.  Scalable inference for crossed random effects models , 2018, Biometrika.

[26]  William J. Browne,et al.  An illustration of the use of reparameterisation methods for improving MCMC efficiency in crossed random effect models , 2004 .

[27]  Catalina A. Vallejos,et al.  BASiCS: Bayesian Analysis of Single-Cell Sequencing Data , 2015, PLoS Comput. Biol..

[28]  Andrew Gelman,et al.  The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo , 2011, J. Mach. Learn. Res..

[29]  Pascal Wild,et al.  Fitting Bayesian multiple random effects models , 1996, Stat. Comput..

[30]  S. Mukherjee,et al.  Approximations of Markov Chains and Bayesian Inference , 2015 .

[31]  Omiros Papaspiliopoulos,et al.  A note on MCMC for nested multilevel regression models via belief propagation , 2017 .

[32]  B. Carlin,et al.  Measures of Bayesian learning and identifiability in hierarchical models , 2006 .

[33]  Y. Amit.  On rates of convergence of stochastic relaxation for Gaussian and non-Gaussian distributions , 1991 .

[34]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[35]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[36]  Jun S. Liu,et al.  Parameter Expansion for Data Augmentation , 1999 .

[37]  Jun S. Liu,et al.  Generalised Gibbs sampler and multigrid Monte Carlo for Bayesian computation , 2000 .