Fast and Accurate Binary Response Mixed Model Analysis via Expectation Propagation

Abstract Expectation propagation is a general prescription for approximation of integrals in statistical inference problems. Its literature is mainly concerned with Bayesian inference scenarios. However, expectation propagation can also be used to approximate integrals arising in frequentist statistical inference. We focus on likelihood-based inference for binary response mixed models and show that fast and accurate quadrature-free inference can be realized for the probit link case with multivariate random effects and higher levels of nesting. The approach is supported by asymptotic calculations in which expectation propagation is seen to provide consistent estimation of the exact likelihood surface. Numerical studies reveal the availability of fast, highly accurate and scalable methodology for binary mixed model analysis.

[1]  P. Diggle,et al.  Analysis of Longitudinal Data. , 1997 .

[2]  Matt P. Wand,et al.  On expectation propagation for generalised, linear and mixed models , 2018 .

[3]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[4]  Subhash R. Lele,et al.  Estimability and Likelihood Inference for Generalized Linear Mixed Models Using Data Cloning , 2010 .

[5]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[6]  D. Bates,et al.  Mixed-Effects Models in S and S-PLUS , 2001 .

[7]  Fiona Steele,et al.  Immunization uptake in rural Bangladesh : a multilevel analysis , 1996 .

[8]  J. Magnus,et al.  Matrix Differential Calculus with Applications in Statistics and Econometrics , 2019, Wiley Series in Probability and Statistics.

[9]  Thomas P. Minka,et al.  Gates: A Graphical Notation for Mixture Models , 2008 .

[10]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[11]  Minjeong Jeon,et al.  A Variational Maximization–Maximization Algorithm for Generalized Linear Mixed Models with Crossed Random Effects , 2017, Psychometrika.

[12]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[13]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[14]  R. Baayen,et al.  Mixed-effects modeling with crossed random effects for subjects and items , 2008 .

[15]  Bradley P. Carlin,et al.  A Sample Reuse Method for Accurate Parametric Empirical Bayes Confidence Intervals , 1991 .

[16]  Thomas P. Minka,et al.  Divergence measures and message passing , 2005 .

[17]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[18]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[19]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[20]  A. Galecki.  JULIAN J. FARAWAY. Extending the Linear Model with R: Generalized Linear, Mixed Effects, and Nonparametric Regression Models, 2nd edition. Boca Raton: CRC Press , 2017 .

[21]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[22]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[23]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[24]  Isabel Molina,et al.  Small Area Estimation: Rao/Small Area Estimation , 2005 .

[25]  Péter Sólymos,et al.  dclone: Data Cloning in R , 2010, R J..

[26]  B. Baltagi,et al.  Econometric Analysis of Panel Data , 2020, Springer Texts in Business and Economics.

[27]  M. Wand,et al.  Simple Marginally Noninformative Prior Distributions for Covariance Matrices , 2013 .

[28]  Harvey Goldstein,et al.  Multilevel Statistical Models: Goldstein/Multilevel Statistical Models , 2010 .

[29]  Larry Nazareth,et al.  A family of variable metric updates , 1977, Math. Program..

[30]  D. Harville.  Matrix Algebra From a Statistician's Perspective , 1998 .

[31]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[32]  Noreen Goldman,et al.  An assessment of estimation procedures for multilevel models with binary responses , 1995 .

[33]  Helen Ogden,et al.  A sequential reduction method for inference in generalized linear mixed models , 2013, 1312.1903.

[34]  John T. Ormerod,et al.  Continued fraction enhancement of Bayesian computing , 2012 .

[35]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[36]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[37]  Guillaume P. Dehaene,et al.  Expectation propagation in the large data limit , 2015, 1503.08060.

[38]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[39]  S. Puntanen.  Generalized, Linear, and Mixed Models, Second Edition by Charles E. McCulloch, Shayle R. Searle, John M. Neuhaus , 2009 .

[40]  H. Rue,et al.  Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations , 2009 .

[41]  C. Borror.  Generalized Linear Models and Extensions, Second Edition , 2008 .