Empirical Bayes vs. Fully Bayes Variable Selection

Abstract For the problem of variable selection for the normal linear model, fixed penalty selection criteria such as AIC, C p , BIC and RIC correspond to the posterior modes of a hierarchical Bayes model for various fixed hyperparameter settings. Adaptive selection criteria obtained by empirical Bayes estimation of the hyperparameters have been shown by George and Foster [2000. Calibration and Empirical Bayes variable selection. Biometrika 87(4), 731–747] to improve on these fixed selection criteria. In this paper, we study the potential of alternative fully Bayes methods, which instead margin out the hyperparameters with respect to prior distributions. Several structured prior formulations are considered for which fully Bayes selection and estimation methods are obtained. Analytical and simulation comparisons with empirical Bayes counterparts are studied.

[1]  James O. Berger,et al.  Objective Bayesian Methods for Model Selection: Introduction and Comparison , 2001 .

[2]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[3]  Colin L. Mallows,et al.  Some Comments on Cp , 2000, Technometrics.

[4]  W. Strawderman.  Proper Bayes Minimax Estimators of the Multivariate Normal Mean , 1971 .

[5]  I. Johnstone,et al.  Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences , 2004, math/0410088.

[6]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[7]  C. Mallows.  More comments on C p , 1995 .

[8]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[9]  A. Zellner,et al.  Basic Issues in Econometrics. , 1986 .

[10]  Thomas S. Shively,et al.  Variable Selection : Empirical Bayes vs . Fully Bayes , 2002 .

[11]  M. Clyde,et al.  Flexible empirical Bayes estimation for wavelets , 2000 .

[12]  Dean Phillips Foster,et al.  Calibration and Empirical Bayes Variable Selection , 1997 .

[13]  C. L. Mallows.  Some comments on C_p , 1973 .

[14]  Dean P. Foster,et al.  The risk inflation criterion for multiple regression , 1994 .

[15]  A. Zellner,et al.  Posterior odds ratios for selected regression hypotheses , 1980 .

[16]  I. Johnstone,et al.  Wavelet Shrinkage: Asymptopia? , 1995 .

[17]  M. Clyde,et al.  Model Uncertainty , 2003 .

[18]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[19]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[20]  C. L. Mallows.  Some Comments onCp , 1973 .

[21]  Seymour Geisser,et al.  On Prior Distributions for Binary Trials , 1984 .

[22]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[23]  G. Casella,et al.  Objective Bayesian Variable Selection , 2006 .

[24]  Edward I. George,et al.  The Practical Implementation of Bayesian Model Selection , 2001 .