Average (E)BIC-like Criteria for Bayesian Model Selection

Markov chain Monte Carlo (MCMC) has been an indispensable tool for Bayesian analysis of complex statistical models even for high-dimensional problems. However, there still lacks a consistent criterion for selecting models based on the outputs of MCMC. The existing deviance information criterion (DIC) is known to be inconsistent and non-invariant for reparameterization. This paper proposes an Average BIC-like (ABIC) model selection criterion and an Average EBIC-like (AEBIC) model selection criterion for low and high-dimensional problems, respectively; establishes their consistency under mild conditions; and illustrates their applications using generalized linear models. The proposed criteria overcome shortcomings of DIC. The numerical results indicate that the proposed criteria can significantly outperform DIC as well as the MLE-based criteria, such as AIC, BIC and EBIC, in terms of model selection accuracy.

[1]  H. Akaike.  A new look at the statistical model identification , 1974 .

[2]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[3]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[4]  R. Nishii.  Asymptotic Properties of Criteria for Selection of Variables in Multiple Regression , 1984 .

[5]  D. Haughton.  On the Choice of a Model to Fit Data from an Exponential Family , 1988 .

[6]  Adrian E. Raftery,et al.  Bayes factors and model uncertainty , 1995 .

[7]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[8]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[9]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[10]  E. Lander,et al.  Gene expression correlates of clinical prostate cancer behavior. , 2002, Cancer cell.

[11]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[12]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[13]  Peng Zhao,et al.  On Model Selection Consistency of Lasso , 2006, J. Mach. Learn. Res..

[14]  Wenxin Jiang.  Bayesian variable selection for high dimensional generalized linear models : Convergence rates of the fitted densities , 2007, 0710.3458.

[15]  R. Carroll,et al.  Stochastic Approximation in Monte Carlo Computation , 2007 .

[16]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[17]  Jiahua Chen,et al.  Extended Bayesian information criteria for model selection with large model spaces , 2008 .

[18]  B. Efron.  Empirical Bayes Estimates for Large-Scale Prediction Problems , 2009, Journal of the American Statistical Association.

[19]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[20]  Cun-Hui Zhang.  Nearly unbiased variable selection under minimax concave penalty , 2010, 1002.4734.

[21]  Sumio Watanabe,et al.  Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory , 2010, J. Mach. Learn. Res..

[22]  X. Huo,et al.  Complexity of penalized likelihood estimation , 2010 .

[23]  Zehua Chen,et al.  Selection Consistency of EBIC for GLIM with Non-canonical Links and Diverging Number of Parameters , 2011, 1112.2815.

[24]  M. Drton,et al.  Bayesian model choice and information criteria in sparse generalized linear models , 2011, 1112.5635.

[25]  V. Johnson,et al.  Bayesian Model Selection in High-Dimensional Settings , 2012, Journal of the American Statistical Association.

[26]  Zehua Chen,et al.  EXTENDED BIC FOR SMALL-n-LARGE-P SPARSE GLM , 2012 .

[27]  F. Liang,et al.  Bayesian Subset Modeling for High-Dimensional Generalized Linear Models , 2013 .

[28]  N. G. Best,et al.  The deviance information criterion: 12 years on , 2014 .

[29]  F. Liang,et al.  A split‐and‐merge Bayesian variable selection approach for ultrahigh dimensional regression , 2015 .

[30]  F. Liang,et al.  High-Dimensional Variable Selection With Reciprocal L1-Regularization , 2015 .

[31]  Hao Yin,et al.  Strong NP-Hardness Result for Regularized $L_q$-Minimization Problems with Concave Penalty Functions , 2015, ArXiv.

[32]  Kean Ming Tan,et al.  Laplace Approximation in High-Dimensional Bayesian Regression , 2015, 1503.08337.

[33]  Minsuk Shin,et al.  Scalable Bayesian Variable Selection Using Nonlocal Prior Densities in Ultrahigh-dimensional Settings. , 2015, Statistica Sinica.