Prior Distributions for Objective Bayesian Analysis

We provide a review of prior distributions for objective Bayesian analysis. We start by examining some foundational issues and then organize our exposition into priors for: i) estimation or prediction; ii) model selection; iii) highdimensional models. With regard to i), we present some basic notions, and then move to more recent contributions on discrete parameter space, hierarchical models, nonparametric models, and penalizing complexity priors. Point ii) is the focus of this paper: it discusses principles for objective Bayesian model comparison, and singles out some major concepts for building priors, which are subsequently illustrated in some detail for the classic problem of variable selection in normal linear models. We also present some recent contributions in the area of objective priors on model space. With regard to point iii) we only provide a short summary of some default priors for high-dimensional models, a rapidly growing area of research.

[1]  C. Robert,et al.  Model choice in generalised linear models: A Bayesian approach via Kullback-Leibler projections , 1998 .

[2]  Roberta Paroli,et al.  Objective Bayesian Comparison of Constrained Analysis of Variance Models , 2013, Psychometrika.

[3]  S. Walker,et al.  On a Global Objective Prior from Score Rules , 2017, 1706.00599.

[4]  Dongchu Sun,et al.  Objective Bayesian analysis for a capture–recapture model , 2014 .

[5]  Bin Yu,et al.  Model Selection and the Principle of Minimum Description Length , 2001 .

[6]  F. Girón,et al.  Comparison of Bayesian objective procedures for variable selection in linear regression , 2008 .

[7]  M. J. Bayarri,et al.  Generalization of Jeffreys divergence‐based priors for Bayesian hypothesis testing , 2008, 0801.4224.

[8]  James O. Berger,et al.  Objective Bayesian Methods for Model Selection: Introduction and Comparison , 2001 .

[9]  M. J. Bayarri,et al.  Objective Bayesian Analysis of Multiple Changepoints for Linear Models , 2006 .

[10]  岩城 克明 Posterior Expected Marginal Likelihood for Testing Hypotheses , 1997 .

[11]  Dimitris Fouskakis,et al.  Limiting behavior of the Jeffreys power-expected-posterior Bayes factor in Gaussian linear models , 2013, Brazilian Journal of Probability and Statistics.

[12]  A. O'Hagan,et al.  Fractional Bayes factors for model comparison , 1995 .

[13]  Merlise A. Clyde,et al.  Mixtures of g-Priors in Generalized Linear Models , 2015, Journal of the American Statistical Association.

[14]  Eric-Jan Wagenmakers,et al.  Editors’ introduction to the special issue “Bayes factors for testing hypotheses in psychological research : Practical relevance and new developments” , 2016 .

[15]  James G. Scott,et al.  Objective Bayesian model selection in Gaussian graphical models , 2009 .

[16]  S. Walker,et al.  On the mathematics of the Jeffreys–Lindley paradox , 2015, 1503.04098.

[17]  Steffen L. Lauritzen,et al.  Compatible prior distributions , 2000 .

[18]  Riten Mitra,et al.  Bayesian Nonparametric Inference - Why and How. , 2013, Bayesian analysis.

[19]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[20]  J. Bernardo,et al.  Bayesian Hypothesis Testing: a Reference Approach , 2002 .

[21]  E. Moreno,et al.  Intrinsic Priors for Objective Bayesian Model Selection , 2014 .

[22]  On the Independence Jeffreys prior for skew-symmetric models with applications , 2013, 1307.6442.

[23]  S. Walker,et al.  An Objective Bayesian Criterion to Determine Model Prior Probabilities , 2015 .

[24]  Yuzo Maruyama,et al.  Posterior Odds with a Generalized Hyper-g-Prior , 2014 .

[25]  Minerva Mukhopadhyay,et al.  A mixture of g-priors for variable selection when the number of regressors grows with the sample size , 2015, 1504.03796.

[26]  I. Good,et al.  Probability and the Weighting of Evidence. , 1951 .

[27]  Clemens Elster,et al.  Objective Bayesian Inference for a Generalized Marginal Random Effects Model , 2016 .

[28]  J. S. Rao,et al.  Spike and slab variable selection: Frequentist and Bayesian strategies , 2005, math/0505633.

[29]  G. Casella,et al.  Objective Bayesian Variable Selection , 2006 .

[30]  James G. Scott,et al.  The horseshoe estimator for sparse signals , 2010 .

[31]  Veronika Rockova,et al.  EMVS: The EM Approach to Bayesian Variable Selection , 2014 .

[32]  Dimitris Fouskakis,et al.  Power-Conditional-Expected Priors: Using g-Priors With Random Imaginary Data for Variable Selection , 2013, 1307.2449.

[33]  Edward I. George,et al.  Dilution priors: Compensating for model space redundancy , 2010 .

[34]  George Casella,et al.  Testing equality of regression coefficients in heteroscedastic normal regression models , 2005 .

[35]  C. D. Litton,et al.  Theory of Probability (3rd Edition) , 1984 .

[36]  Scott C Schmidler,et al.  BAYESIAN MODEL SEARCH AND MULTILEVEL INFERENCE FOR SNP ASSOCIATION STUDIES. , 2009, The annals of applied statistics.

[37]  J. Forster,et al.  The Whetstone and the Alum Block: Balanced Objective Bayesian Comparison of Nested Models for Discrete Data , 2013, 1310.0661.

[38]  J. Berger,et al.  Expected‐posterior prior distributions for model selection , 2002 .

[39]  Larry A. Wasserman,et al.  Iterative Markov Chain Monte Carlo Computation of Reference Priors and Minimax Risk , 2001, UAI.

[40]  James O. Berger,et al.  The Effective Sample Size , 2014 .

[41]  James G. Scott,et al.  Local shrinkage rules, Lévy processes and regularized regression , 2010, 1010.3390.

[42]  Dongchu Sun,et al.  Objective priors for hypothesis testing in one‐way random effects models , 2007 .

[43]  G. Casella,et al.  Consistency of Bayesian procedures for variable selection , 2009, 0904.2978.

[44]  James O. Berger,et al.  Estimating a Product of Means: Bayesian Analysis with Reference Priors , 1989 .

[45]  L. Wasserman,et al.  A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion , 1995 .

[46]  James O. Berger,et al.  Ordered group reference priors with application to the multinomial problem , 1992 .

[47]  V. Johnson,et al.  On the use of non‐local prior densities in Bayesian hypothesis tests , 2010 .

[48]  James O. Berger,et al.  Posterior propriety and admissibility of hyperpriors in normal hierarchical models , 2005, math/0505605.

[49]  Walter Racugno,et al.  Bayesian model selection approach to analysis of variance under heteroscedasticity , 2000 .

[50]  M. Steel,et al.  Mixtures of G-Priors for Bayesian Model Averaging with Economic Application , 2011 .

[51]  D. Draper,et al.  Bayesian variable selection using cost-adjusted BIC, with application to cost-effective measurement of quality of health care , 2009, 0908.2313.

[52]  James G. Scott,et al.  On the half-cauchy prior for a global scale parameter , 2011, 1104.4937.

[53]  E. George,et al.  The Spike-and-Slab LASSO , 2018 .

[54]  Joseph G. Ibrahim,et al.  Power prior distributions for generalized linear models , 2000 .

[55]  S. Walker,et al.  An Objective Approach to Prior Mass Functions for Discrete Parameter Spaces , 2015 .

[56]  J. Berger,et al.  The Intrinsic Bayes Factor for Model Selection and Prediction , 1996 .

[57]  B. Liseo,et al.  Objective Bayesian Analysis of Skew‐t Distributions , 2013 .

[58]  Christian P. Robert,et al.  The Bayesian choice , 1994 .

[59]  Kathryn Chaloner,et al.  Bayesian Experimental Design for Nonlinear Mixed‐Effects Models with Application to HIV Dynamics , 2004, Biometrics.

[60]  Elías Moreno,et al.  Objective Bayesian methods for one-sided testing , 2005 .

[61]  J. Neyman,et al.  INADMISSIBILITY OF THE USUAL ESTIMATOR FOR THE MEAN OF A MULTIVARIATE NORMAL DISTRIBUTION , 2005 .

[62]  B. Liseo,et al.  Objective priors for causal AR(p) with partial autocorrelations , 2013 .

[63]  Guido Consonni,et al.  Moment priors for Bayesian model choice with applications to directed acyclic graphs , 2011 .

[64]  Walter Racugno,et al.  An Intrinsic Limiting Procedure for Model Selection and Hypotheses Testing , 1998 .

[65]  Leonhard Held,et al.  Hyper-$g$ priors for generalized linear models , 2010, 1008.1550.

[66]  John D. Storey The positive false discovery rate: a Bayesian interpretation and the q-value , 2003 .

[67]  Judith Rousseau,et al.  On adaptive posterior concentration rates , 2013, 1305.5270.

[68]  M. J. Bayarri,et al.  Extending conventional priors for testing general hypotheses in linear models , 2007 .

[69]  D. Spiegelhalter,et al.  Bayes Factors for Linear and Log‐Linear Models with Vague Prior Information , 1982 .

[70]  Ioannis Ntzoufras,et al.  Power-Expected-Posterior Priors for Variable Selection in Gaussian Linear Models , 2013, 1307.2442.

[71]  M. Clyde,et al.  Mixtures of g Priors for Bayesian Variable Selection , 2008 .

[72]  N. Pillai,et al.  Dirichlet–Laplace Priors for Optimal Shrinkage , 2014, Journal of the American Statistical Association.

[73]  George Casella,et al.  Assessing Robustness of Intrinsic Tests of Independence in Two-Way Contingency Tables , 2009 .

[74]  I. Ntzoufras,et al.  Bayesian spatio-temporal epidemic models with applications to sheep pox , 2014, 1403.1783.

[75]  Leonhard Held,et al.  Objective Bayesian Model Selection in Generalized Additive Models With Penalized Splines , 2015 .

[76]  Jayanta K. Ghosh,et al.  Asymptotic Properties of Bayes Risk for the Horseshoe Prior , 2013 .

[77]  Aki Vehtari,et al.  On the Hyperprior Choice for the Global Shrinkage Parameter in the Horseshoe Prior , 2016, AISTATS.

[78]  Dongchu Sun,et al.  PROPRIETY OF POSTERIORS WITH IMPROPER PRIORS IN HIERARCHICAL LINEAR MIXED MODELS , 2001 .

[79]  J. Berger,et al.  Choice of hierarchical priors: admissibility in estimation of normal means , 1996 .

[80]  S. MacEachern,et al.  Minimally informative prior distributions for non‐parametric Bayesian analysis , 2010 .

[81]  J. Berger,et al.  Optimal predictive model selection , 2004, math/0406464.

[82]  Simon Guillotte,et al.  Nonparametric Bayesian Estimation of a Bivariate Copula Using the Jeffreys Prior , 2009, 0908.2372.

[83]  Malay Ghosh,et al.  Objective Priors: An Introduction for Frequentists , 2011, 1108.2120.

[84]  E. Wagenmakers,et al.  A Default Bayesian Hypothesis Test for ANOVA Designs , 2012 .

[85]  S. MacEachern,et al.  Local-Mass Preserving Prior Distributions for Nonparametric Bayesian Models , 2014 .

[86]  D. Spitzner Risk‐reducing shrinkage estimation for generalized linear models , 2005 .

[87]  Christopher D. Chambers,et al.  Redefine statistical significance , 2017, Nature Human Behaviour.

[88]  M. Plummer Penalized loss functions for Bayesian model comparison. , 2008, Biostatistics.

[89]  G. Casella,et al.  CONSISTENCY OF OBJECTIVE BAYES FACTORS AS THE MODEL DIMENSION GROWS , 2010, 1010.3821.

[90]  M. J. Bayarri,et al.  Criteria for Bayesian model choice with application to variable selection , 2012, 1209.5240.

[91]  Judith Rousseau,et al.  Posterior concentration rates for infinite dimensional exponential families , 2012 .

[92]  Yuzo Maruyama,et al.  Fully Bayes factors with a generalized g-prior , 2008, 0801.4410.

[93]  Mathieu Kessler,et al.  On intrinsic priors for nonnested models , 2004 .

[94]  Jukka Corander,et al.  A Bayesian Approach to Modelling Graphical Vector Autoregressions , 2006 .

[95]  James O. Berger,et al.  Noninformative priors for inferences in exponential regression models , 1991 .

[96]  J. Berger The case for objective Bayesian analysis , 2006 .

[97]  J. Bernardo Reference Posterior Distributions for Bayesian Inference , 1979 .

[98]  Luis R. Pericchi,et al.  Model Selection and Hypothesis Testing based on Objective Probabilities and Bayes Factors , 2005 .

[99]  Luis R. Pericchi,et al.  On the robustness of the intrinsic Bayes factor for nested models , 1996 .

[100]  L. Pericchi,et al.  The Scaled Beta2 Distribution as a Robust Prior for Scales , 2017 .

[101]  D. Spitzner Neutral-data comparisons for Bayesian testing , 2011 .

[102]  James O. Berger,et al.  Objective Priors for Discrete Parameter Spaces , 2012 .

[103]  P. Dellaportas,et al.  Bayesian variable and link determination for generalised linear models , 2003 .

[104]  J. Berger Statistical Decision Theory and Bayesian Analysis , 1988 .

[105]  Rubin Herman,et al.  A WEAK SYSTEM OF AXIOMS FOR "RATIONAL" BEHAVIOR AND THE NONSEPARABILITY OF UTILITY FROM PRIOR , 1987 .

[106]  M. Stone,et al.  Marginalization Paradoxes in Bayesian and Structural Inference , 1973 .

[107]  James G. Scott,et al.  Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem , 2010, 1011.2333.

[108]  Guido Consonni,et al.  Objective Bayes Factors for Gaussian Directed Acyclic Graphical Models , 2012 .

[109]  Nicholas G. Polson,et al.  Default Bayesian Analysis with Global-Local Shrinkage Priors , 2015, 1510.03516.

[110]  Fulvio Spezzaferri,et al.  Alternative Bayes factors for model selection , 1997 .

[111]  A. V. D. Vaart,et al.  Needles and Straw in a Haystack: Posterior concentration for possibly sparse sequences , 2012, 1211.1197.

[112]  Dongchu Sun,et al.  Objective priors for the bivariate normal model , 2008, 0804.0987.

[113]  D. Madigan,et al.  A method for simultaneous variable selection and outlier identification in linear regression , 1996 .

[114]  B. Liseo,et al.  A default Bayesian test for the number of components in a mixture , 2003 .

[115]  S. MacEachern,et al.  Block Hyper-g Priors in Bayesian Regression , 2014, 1406.6419.

[116]  Dongchu Sun,et al.  Intrinsic Priors for Model Selection Using an Encompassing Model with Applications to Censored Failure Time Data , 2000, Lifetime data analysis.

[117]  F. Girón,et al.  Intrinsic priors for model comparison in multivariate normal regression , 2011 .

[118]  Jianhua Hu,et al.  Bayesian model selection using test statistics , 2009, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[119]  Julia Mortera,et al.  Default Bayes Factors for Nonnested Hypothesis Testing , 1999 .

[120]  Monica Musio,et al.  Bayesian Model Selection Based on Proper Scoring Rules , 2014, 1409.5291.

[121]  Claudio Fuentes,et al.  Model Space Priors for Objective Sparse Bayesian Regression , 2015, 1511.04745.

[122]  I. Ntzoufras,et al.  Information consistency of the Jeffreys power-expected-posterior prior in Gaussian linear models , 2017 .

[123]  Yingbo Li,et al.  Bayesian Hierarchical Models for Model Choice , 2013 .

[124]  Cristiano Villa,et al.  Model prior distribution for variable selection in linear regression models , 2015, 1512.08077.

[125]  G. Casella,et al.  Inference From Intrinsic Bayes’ Procedures Under Model Selection and Uncertainty , 2014 .

[126]  Valen E. Johnson,et al.  Bayes factors based on test statistics , 2005 .

[127]  Edward I. George,et al.  Empirical Bayes vs. Fully Bayes Variable Selection , 2008 .

[128]  J. Bernardo,et al.  THE FORMAL DEFINITION OF REFERENCE PRIORS , 2009, 0904.0156.

[129]  Juho Piironen,et al.  Uncertainty Quantification for the Horseshoe (with Discussion) comment , 2017 .

[130]  J. Griffin,et al.  Inference with normal-gamma prior distributions in regression problems , 2010 .

[131]  Merlise A. Clyde,et al.  Bayesian Model Averaging in the M-Open Framework , 2013 .

[132]  E. George,et al.  Journal of the American Statistical Association is currently published by American Statistical Association. , 2007 .

[133]  S. Walker,et al.  A cautionary note on the discrete uniform prior for the binomial N: comment , 2014 .

[134]  Thiago G. Martins,et al.  Penalising Model Component Complexity: A Principled, Practical Approach to Constructing Priors , 2014, 1403.4630.

[135]  I. Good Significance Tests in Parallel and in Series , 1958 .

[136]  Stefano Peluso,et al.  Objective Bayes Covariate‐Adjusted Sparse Graphical Model Selection , 2015, 1510.02245.

[137]  A. Dawid,et al.  Posterior Model Probabilities , 2011 .

[138]  C. Robert,et al.  Bayesian Modeling Using WinBUGS , 2009 .

[139]  Elías Moreno,et al.  Bayes factors for intrinsic and fractional priors in nested models. Bayesian robustness , 1997 .

[140]  Ming-Hui Chen,et al.  Objective Bayesian Inference for Bilateral Data. , 2015, Bayesian analysis.

[141]  Dean P. Foster,et al.  The risk inflation criterion for multiple regression , 1994 .

[142]  M. Steel,et al.  Benchmark Priors for Bayesian Model Averaging , 2001 .

[143]  Geoffrey Gregory,et al.  Foundations of Statistical Inference , 1973 .

[144]  L. Wasserman,et al.  The Selection of Prior Distributions by Formal Rules , 1996 .

[145]  A. P. Dawid,et al.  Invariant Prior Distributions , 2006 .

[146]  Jonathan J. Forster,et al.  Default Bayesian model determination methods for generalised linear mixed models , 2010, Comput. Stat. Data Anal..

[147]  Matteo Ruggiero,et al.  Are Gibbs-Type Priors the Most Natural Generalization of the Dirichlet Process? , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[148]  Oluwasanmi Koyejo,et al.  False Discovery Rate Smoothing , 2014, Journal of the American Statistical Association.

[149]  Serena Arima,et al.  Objective Bayesian analysis of a measurement error small area model , 2012 .

[150]  Radford M. Neal Transferring Prior Information Between Models Using Imaginary Data , 2001 .

[151]  P. Gustafson,et al.  Conservative prior distributions for variance parameters in hierarchical models , 2006 .

[152]  Haavard Rue,et al.  Penalised Complexity Priors for Stationary Autoregressive Processes , 2016, 1608.08941.

[153]  Jukka Corander,et al.  Bayesian assessment of dimensionality in reduced rank regression , 2004 .

[154]  G. Consonni,et al.  Testing Hardy–Weinberg equilibrium: An objective Bayesian analysis , 2011, Statistics in medicine.

[155]  J. Berger,et al.  Objective Bayesian Analysis of Spatially Correlated Data , 2001 .

[156]  Raffaello Seri,et al.  Estimation in Discrete Parameter Models , 2012, 1207.5653.

[157]  M. Villani Fractional Bayesian Lag Length Inference in Multivariate Autoregressive Processes , 2001 .

[158]  M. Yuan,et al.  Efficient Empirical Bayes Variable Selection and Estimation in Linear Models , 2005 .

[159]  Stephen G. Walker,et al.  On Priors With a Kullback–Leibler Property , 2004 .

[160]  Donatello Telesca,et al.  Nonlocal Priors for High-Dimensional Estimation , 2014, Journal of the American Statistical Association.

[161]  J. M. Pérez,et al.  Development of expected posterior prior distributions for model comparisons , 1998 .

[162]  G. Casella,et al.  Objective Bayes model selection in probit models , 2012, Statistics in medicine.

[163]  J. Cano,et al.  Integral Priors and Constrained Imaginary Training Samples for Nested and Non-nested Bayesian Model Comparison , 2013 .

[164]  Dimitris Fouskakis,et al.  Power-Expected-Posterior Priors for Generalized Linear Models , 2015, Bayesian Analysis.

[165]  I. Castillo,et al.  Empirical Bayes analysis of spike and slab posterior distributions , 2018, 1801.01696.

[166]  D. Lindley A STATISTICAL PARADOX , 1957 .

[167]  Helio S. Migon,et al.  Objective Bayesian analysis for the Student-t regression model , 2008 .

[168]  Min Wang,et al.  Mixtures of $g$-Priors for Analysis of Variance Models with a Diverging Number of Parameters , 2017 .

[169]  V. Johnson UNIFORMLY MOST POWERFUL BAYESIAN TESTS. , 2013, Annals of statistics.

[170]  Petros Dellaportas,et al.  Joint Specification of Model Space and Parameter Space Prior Distributions , 2012, 1207.5651.

[171]  M. Steel,et al.  Objective Bayesian Survival Analysis Using Shape Mixtures of Log-Normal Distributions , 2015 .

[172]  Dean Phillips Foster,et al.  Calibration and Empirical Bayes Variable Selection , 1997 .

[173]  L. Pericchi,et al.  BAYES FACTORS AND MARGINAL DISTRIBUTIONS IN INVARIANT SITUATIONS , 2016 .

[174]  Todd A. Kuffner,et al.  Bayes factor consistency , 2016, 1607.00292.

[175]  A. Bhattacharya,et al.  Bayesian fractional posteriors , 2016, The Annals of Statistics.

[176]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[177]  Edward I. George,et al.  The Practical Implementation of Bayesian Model Selection , 2001 .

[178]  Guido Consonni,et al.  Tests Based on Intrinsic Priors for the Equality of Two Correlated Proportions , 2008 .

[179]  J. Ibrahim,et al.  Power prior distributions for regression models , 2000 .

[180]  R. Mukerjee,et al.  Probability Matching Priors: Higher Order Asymptotics , 2004 .

[181]  A. V. D. Vaart,et al.  BAYESIAN LINEAR REGRESSION WITH SPARSE PRIORS , 2014, 1403.0735.

[182]  Leonhard Held,et al.  Approximate Bayesian Model Selection with the Deviance Statistic , 2013, 1308.6780.

[183]  C. Robert,et al.  Testing hypotheses via a mixture estimation model , 2014, 1412.2044.

[184]  A. Gelman,et al.  Beyond subjective and objective in statistics , 2015, 1508.05453.

[185]  Aki Vehtari,et al.  Sparsity information and regularization in the horseshoe and other shrinkage priors , 2017, 1707.01694.

[186]  E. Moreno,et al.  Objective Testing Procedures in Linear Models: Calibration of the p‐values , 2006 .

[187]  L. Brown Admissible Estimators, Recurrent Diffusions, and Insoluble Boundary Value Problems , 1971 .

[188]  Ioannis Ntzoufras,et al.  Bayesian Analysis of Marginal Log-Linear Graphical Models for Three Way Contingency Tables , 2008 .

[189]  Guido Consonni,et al.  Compatibility of prior specifications across linear models , 2008, 1102.2981.

[190]  Jukka Corander,et al.  Learning Gaussian graphical models with fractional marginal pseudo-likelihood , 2016, Int. J. Approx. Reason..

[191]  Guido Consonni,et al.  Objective Bayesian Search of Gaussian Directed Acyclic Graphical Models for Ordered Variables with Non‐Local Priors , 2013, Biometrics.

[192]  V. Rocková,et al.  Bayesian estimation of sparse signals with a continuous spike-and-slab prior , 2018 .