Objective Bayes Criteria for Variable Selection

Variable selection typically involves choosing among a large number of models, so that fast computation of Bayes factors is highly desirable. This desideratum has made common practice the use of g-priors and Laplace expansions, specially in large dimensions. It is well known, however, that priors with heavier tails often result in better performance for model selection. In this thesis, we use the Conventional approach of Jeffreys (1961) and generalize some ideas in Strawderman (1971, 1973) and Berger (1976, 1980, 1985) to propose a prior distribution for vari- able selection. We show that this choice is, to the best of our knowl- edge, the first proposal for variable selection which is fully justified from a theoretical point of view. This justification is heavily based on the invariance ideas in Berger et al. (1998). Moreover, it has Student-like tails and many optimal properties for model selection. It also generalizes previous proposals in the literature. In addition, for specific choices of the hyper-parameters, it produces closed-form marginal likelihoods (and hence, Bayes factors). We demonstrate its behavior in a couple of small problems and in a couple of large, but enumerable, ones.

[1]  C. Robert,et al.  Model choice in generalised linear models: A Bayesian approach via Kullback-Leibler projections , 1998 .

[2]  James O. Berger,et al.  Objective Bayesian Methods for Model Selection: Introduction and Comparison , 2001 .

[3]  I. Guttman.  Linear models : an introduction , 1982 .

[4]  M. J. Bayarri,et al.  P Values for Composite Null Models , 2000 .

[5]  B. Carlin,et al.  Bayesian Model Choice Via Markov Chain Monte Carlo Methods , 1995 .

[6]  E. George,et al.  APPROACHES FOR BAYESIAN VARIABLE SELECTION , 1997 .

[7]  Edward E. Leamer,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[8]  Purushottam W. Laud,et al.  A Predictive Approach to the Analysis of Designed Experiments , 1994 .

[9]  M. J. Bayarri,et al.  Bayesian measures of surprise for outlier detection , 2003 .

[10]  James G. Scott,et al.  Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem , 2010, 1011.2333.

[11]  J. Berger.  A Robust Generalized Bayes Estimator and Confidence Region for a Multivariate Normal Mean , 1980 .

[12]  M. J. Bayarri,et al.  Extending conventional priors for testing general hypotheses in linear models , 2007 .

[13]  Gonzalo García-Donato Layrón.  Factores bayes y factores bayes convencionales: algunos aspectos relevantes , 2003 .

[14]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[15]  Anthony O'Hagan.  Probability: Methods And Measurement , 1988 .

[16]  Chuhsing Kate Hsiao,et al.  Approximate Bayes Factors When a Mode Occurs on the Boundary , 1997 .

[17]  A. Zellner,et al.  Basic Issues in Econometrics. , 1986 .

[18]  M. J. Bayarri,et al.  Bayesian Checking of the Second Levels of Hierarchical Models. Rejoinder. , 2007, 0802.0743.

[19]  J. Berger.  Admissible Minimax Estimation of a Multivariate Normal Mean with Arbitrary Quadratic Loss , 1976 .

[20]  H. Jeffreys,et al.  The Theory of Probability , 1896 .

[21]  Alan E. Gelfand,et al.  Model choice: A minimum posterior predictive loss approach , 1998, AISTATS.

[22]  M. L. Eaton.  Group invariance applications in statistics , 1989 .

[23]  Anne Lohrli.  Chapman and Hall , 1985 .

[24]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[25]  M. Steel,et al.  Benchmark Priors for Bayesian Model Averaging , 2001 .

[26]  J. Friedman,et al.  Estimating Optimal Transformations for Multiple Regression and Correlation. , 1985 .

[27]  R. Kass.  Bayes Factors in Practice , 1993 .

[28]  P. Rousseeuw,et al.  Wiley Series in Probability and Mathematical Statistics , 2005 .

[29]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[30]  J. Richard,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[31]  R. Kass,et al.  Approximate Bayes Factors and Orthogonal Parameters, with Application to Testing Equality of Two Binomial Proportions , 1992 .

[32]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[33]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[34]  J. Berger.  The case for objective Bayesian analysis , 2006 .

[35]  J. Ghosh,et al.  Approximations and consistency of Bayes factors as model dimension grows , 2003 .

[36]  Adrian E. Raftery,et al.  Bayesian model averaging: a tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors , 1999 .

[37]  S. James Press,et al.  Subjective and objective Bayesian statistics : principles, models, and applications , 2003 .

[38]  Joseph G. Ibrahim,et al.  Criterion-based methods for Bayesian model assessment , 2001 .

[39]  J. Bernardo,et al.  THE FORMAL DEFINITION OF REFERENCE PRIORS , 2009, 0904.0156.

[40]  Wayne S. Smith,et al.  Interactive Elicitation of Opinion for a Normal Linear Model , 1980 .

[41]  Solomon Kullback,et al.  Information Theory and Statistics , 1960 .

[42]  G. Casella,et al.  Objective Bayesian Variable Selection , 2006 .

[43]  G. Casella,et al.  Consistency of Bayesian procedures for variable selection , 2009, 0904.2978.

[44]  James O. Berger,et al.  Posterior model probabilities via path‐based pairwise priors , 2005 .

[45]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[46]  William E. Strawderman,et al.  Proper Bayes Minimax Estimators of the Multivariate Normal Mean Vector for the Case of Common Unknown Variances , 1973 .

[47]  A. Atkinson.  Subset Selection in Regression , 1992 .

[48]  E. George.  The Variable Selection Problem , 2000 .

[49]  W. Strawderman.  Proper Bayes Minimax Estimators of the Multivariate Normal Mean , 1971 .

[50]  D. Madigan,et al.  Bayesian Model Averaging for Linear Regression Models , 1997 .

[51]  Joel B. Greenhouse,et al.  [Investigating Therapies of Potentially Great Benefit: ECMO]: Comment: A Bayesian Perspective , 1989 .

[52]  H. H. Steinour,et al.  Effect of Composition of Portland Cement on Heat Evolved during hardening , 1932 .

[53]  Purushottam W. Laud,et al.  Predictive Model Selection , 1995 .

[54]  A. O'Hagan,et al.  Fractional Bayes factors for model comparison , 1995 .

[55]  R. Chattamvelli,et al.  On the doubly noncentral F distribution , 1995 .

[56]  M. J. Bayarri,et al.  Generalization of Jeffreys divergence‐based priors for Bayesian hypothesis testing , 2008, 0801.4224.

[57]  J. Berger,et al.  Optimal predictive model selection , 2004, math/0406464.

[58]  S. R. Searle.  Linear Models , 1971 .

[59]  A. W. Kemp,et al.  Kendall's Advanced Theory of Statistics. , 1994 .

[60]  Na Li,et al.  Simple Parallel Statistical Computing in R , 2007 .

[61]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[62]  I. Ehrlich.  Participation in Illegitimate Activities: A Theoretical and Empirical Investigation , 1973, Journal of Political Economy.

[63]  Wasserman,et al.  Bayesian Model Selection and Model Averaging. , 2000, Journal of mathematical psychology.

[64]  Milton Abramowitz,et al.  Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables , 1964 .

[65]  A. Zellner,et al.  Posterior odds ratios for selected regression hypotheses , 1980 .

[66]  D. Spiegelhalter,et al.  Bayes Factors for Linear and Log‐Linear Models with Vague Prior Information , 1982 .

[67]  Michael Goldstein,et al.  Subjective Bayesian Analysis: Principles and Practice , 2006 .

[68]  J. Ghosh,et al.  Nonsubjective Bayes testing—an overview , 2002 .