Objective Bayesian inference with proper scoring rules

Standard Bayesian analyses can be difficult to perform when the full likelihood, and consequently the full posterior distribution, is too complex or even impossible to specify or if robustness with respect to data or to model misspecifications is required. In these situations, we suggest to resort to a posterior distribution for the parameter of interest based on proper scoring rules. Scoring rules are loss functions designed to measure the quality of a probability distribution for a random variable, given its observed value. Important examples are the Tsallis score and the Hyvärinen score, which allow us to deal with model misspecifications or with complex models. Also the full and the composite likelihoods are both special instances of scoring rules. The aim of this paper is twofold. Firstly, we discuss the use of scoring rules in the Bayes formula in order to compute a posterior distribution, named SR-posterior distribution, and we derive its asymptotic normality. Secondly, we propose a procedure for building default priors for the unknown parameter of interest that can be used to update the information provided by the scoring rule in the SR-posterior distribution. In particular, a reference prior is obtained by maximizing the average $$\alpha $$α-divergence from the SR-posterior distribution. For $$0 \le |\alpha |<1$$0≤|α|<1, the result is a Jeffreys-type prior that is proportional to the square root of the determinant of the Godambe information matrix associated with the scoring rule. Some examples are discussed.

[1]  Aapo Hyvärinen,et al.  Estimation of Non-Normalized Statistical Models by Score Matching , 2005, J. Mach. Learn. Res..

[2]  S. Walker,et al.  On a Global Objective Prior from Score Rules , 2017, 1706.00599.

[3]  S. Fienberg.  When did Bayesian inference become "Bayesian"? , 2006 .

[4]  John Law,et al.  Robust Statistics—The Approach Based on Influence Functions , 1986 .

[5]  J. Ghosh,et al.  On Divergence Measures Leading to Jeffreys and Other Reference Priors , 2014 .

[6]  L. Ventura,et al.  Default prior distributions from quasi- and quasi-profile likelihoods , 2010 .

[7]  A. Dawid.  The geometry of proper scoring rules , 2007 .

[8]  Rahul Mukerjee,et al.  Bayesian and frequentist confidence intervals arising from empirical-type likelihoods , 2008 .

[9]  Stephen G. Walker,et al.  Bayesian information in an experiment and the Fisher information distance , 2016 .

[10]  S. Lauritzen,et al.  Proper local scoring rules , 2011, 1101.5011.

[11]  James O. Berger,et al.  Objective Priors for Discrete Parameter Spaces , 2012 .

[12]  V. P. Godambe.  An Optimum Property of Regular Maximum Likelihood Estimation , 1960 .

[13]  N. Lazar.  Bayesian empirical likelihood , 2003 .

[14]  J. Bernardo.  Reference Analysis , 2005 .

[15]  Susanne M. Schennach,et al.  Accompanying document to "Point Estimation with Exponentially Tilted Empirical Likelihood" , 2005, math/0512181.

[16]  V. Mameli,et al.  Bootstrap adjustments of signed scoring rule root statistics , 2018, Commun. Stat. Simul. Comput..

[17]  Anthony C. Atkinson,et al.  A Parametric Framework for the Comparison of Methods of Very Robust Regression , 2014, 1405.5040.

[18]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[19]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[20]  Susanne M. Schennach,et al.  Bayesian exponentially tilted empirical likelihood , 2005 .

[21]  A. Dawid,et al.  Minimum Scoring Rule Inference , 2014, 1403.3920.

[22]  J. Berger.  The case for objective Bayesian analysis , 2006 .

[23]  Musio Monica,et al.  Bayesian Inference for directional data through ABC and homogeneous proper scoring rules , 2017 .

[24]  M. C. Jones,et al.  Robust and efficient estimation by minimising a density power divergence , 1998 .

[25]  Jayanta K. Ghosh,et al.  Characterization of priors under which Bayesian and frequentist Barlett corrections are equivalent in the multiparameter case , 1991 .

[26]  Laura Ventura,et al.  Bayesian composite marginal likelihoods , 2011 .

[27]  Monica Musio,et al.  Bayesian Model Selection Based on Proper Scoring Rules , 2014, 1409.5291.

[28]  Alec Stephenson,et al.  An extended Gaussian max-stable process model for spatial extremes , 2009 .

[29]  A. Basu,et al.  Robust Bayes estimation using the density power divergence , 2016 .

[30]  R. Machete.  Contrasting probabilistic scoring rules , 2011, 1112.4530.

[31]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[32]  A. Dawid,et al.  Theory and applications of proper scoring rules , 2014, 1401.0398.

[33]  Richard E. Chandler,et al.  Inference for clustered data using the independence loglikelihood , 2007 .

[34]  James O. Berger,et al.  Overall Objective Priors , 2015, 1504.02689.

[35]  Stephane Heritier,et al.  Robust Methods in Biostatistics , 2009 .

[36]  R. Mukerjee,et al.  Probability Matching Priors: Higher Order Asymptotics , 2004 .

[37]  N. Reid,et al.  AN OVERVIEW OF COMPOSITE LIKELIHOOD METHODS , 2011 .

[38]  Luigi Pace,et al.  ADJUSTING COMPOSITE LIKELIHOOD RATIO STATISTICS , 2009 .

[39]  Aapo Hyvärinen,et al.  Some extensions of score matching , 2007, Comput. Stat. Data Anal..

[40]  Laura Ventura,et al.  Higher-order asymptotics for scoring rules , 2015 .

[41]  Laura Ventura,et al.  Robust likelihood functions in Bayesian inference , 2008 .

[42]  K. Mardia,et al.  Score matching estimators for directional distributions , 2016, 1604.08470.

[43]  Erlis Ruli,et al.  Approximate Bayesian computation with composite score functions , 2013, Stat. Comput..

[44]  Laura Ventura,et al.  An overview of robust methods in medical research , 2012, Statistical methods in medical research.

[45]  Malay Ghosh,et al.  Objective Priors: An Introduction for Frequentists , 2011, 1108.2120.

[46]  J. Bernardo,et al.  THE FORMAL DEFINITION OF REFERENCE PRIORS , 2009, 0904.0156.

[47]  A. Basu,et al.  Robust estimation for independent non-homogeneous observations using density power divergence with applications to linear regression , 2013 .

[48]  C. Tsallis.  Possible generalization of Boltzmann-Gibbs statistics , 1988 .

[49]  J. M. Corcuera,et al.  A Characterization of Monotone and Regular Divergences , 1998 .

[50]  Lu Lin.  Quasi Bayesian likelihood , 2006 .

[51]  M. Ghosh,et al.  A general divergence criterion for prior selection , 2011 .

[52]  Xuming He,et al.  Bayesian empirical likelihood for quantile regression , 2012, 1207.5378.

[53]  A. Davison,et al.  Bayesian Inference from Composite Likelihoods, with an Application to Spatial Extremes , 2009, 0911.5357.

[54]  Steffen L. Lauritzen,et al.  The Geometry of Decision Theory , 2006 .

[55]  Laura Ventura,et al.  Pseudo-Likelihoods for Bayesian Inference , 2016 .

[56]  J. Bernardo.  Reference Posterior Distributions for Bayesian Inference , 1979 .