Detecting and diagnosing prior and likelihood sensitivity with power-scaling

Determining the sensitivity of the posterior to perturbations of the prior and likelihood is an important part of the Bayesian workflow. We introduce a practical and computationally efficient sensitivity analysis approach using importance sampling to estimate properties of posteriors resulting from power-scaling the prior or likelihood. On this basis, we suggest a diagnostic that can indicate the presence of prior-data conflict or likelihood noninformativity and discuss limitations to this power-scaling approach. The approach can be easily included in Bayesian workflows with minimal effort by the model builder and we present an implementation in our new R package priorsense . We further demonstrate the workflow on case studies of real data using models varying in complexity from simple linear models to Gaussian process models.

[1]  Philippe Gagnon,et al.  Robustness Against Conflicting Prior Information in Regression , 2021, Bayesian Analysis.

[2]  Aki Vehtari,et al.  Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison , 2021, Stat. Comput..

[3]  T. Yarkoni,et al.  Bambi: A Simple Interface for Fitting Bayesian Linear Models in Python , 2020, J. Stat. Softw..

[4]  Aki Vehtari,et al.  Using reference models in variable selection , 2020, Computational Statistics.

[5]  Aki Vehtari,et al.  Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming , 2020, Statistics and Computing.

[6]  Haavard Rue,et al.  Quantification of Empirical Determinacy: The Impact of Likelihood Weighting on Posterior Location and Spread in Bayesian Meta-Analysis Estimated with JAGS and INLA , 2021, Bayesian Analysis.

[7]  H. Rue,et al.  Sensitivity and identification quantification by a relative latent model complexity perturbation in Bayesian meta‐analysis , 2021, Biometrical journal. Biometrische Zeitschrift.

[8]  Forough Fazeli Asl,et al.  Measuring Bayesian Robustness Using Rényi Divergence , 2021 .

[9]  N. Banholzer,et al.  Estimating the effects of non-pharmaceutical interventions on the number of new infections with COVID-19 during the first epidemic wave , 2021, medRxiv.

[10]  J. Riou,et al.  Bayesian workflow for disease transmission modeling in Stan , 2020, Statistics in medicine.

[11]  Aki Vehtari,et al.  Uncertainty-aware sensitivity analysis using Rényi divergences , 2019, UAI.

[12]  Aki Vehtari,et al.  Implicitly adaptive importance sampling , 2019, Statistics and Computing.

[13]  Roger W. Johnson Fitting Percentage of Body Fat to Simple Body Measurements: College Women , 1996, Journal of Statistics and Data Science Education.

[14]  S. D. Winter,et al.  The Importance of Prior Sensitivity Analysis in Bayesian Statistics: Demonstrations Using an Interactive Shiny App , 2020, Frontiers in Psychology.

[15]  Aki Vehtari,et al.  Regression and Other Stories , 2020 .

[16]  S. Bhatt,et al.  Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe , 2020, Nature.

[17]  J. Rocklöv,et al.  The reproductive number of COVID-19 is higher compared to SARS coronavirus , 2020, Journal of travel medicine.

[18]  M. Betancourt,et al.  Toward a principled Bayesian workflow in cognitive science. , 2019, Psychological methods.

[19]  Aki Vehtari,et al.  Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC (with Discussion) , 2019, Bayesian Analysis.

[20]  Michael Evans,et al.  Checking for Prior-Data Conflict Using Prior-to-Posterior Divergences , 2016, Statistical Science.

[21]  H. Bondell,et al.  Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior , 2016, Journal of the American Statistical Association.

[22]  Arno Solin,et al.  Hilbert space methods for reduced-rank Gaussian process regression , 2014, Stat. Comput..

[23]  B. Liseo Bayesian Robustness , 2019, Wiley StatsRef: Statistics Reference Online.

[24]  P. Ho Global Robust Bayesian Analysis in Large Models , 2019, Federal Reserve Bank of Richmond Working Papers.

[25]  L. Al-Labadi,et al.  Measuring Bayesian Robustness Using Rényi's Divergence and Relationship with Prior-Data Conflict , 2019, 1905.05945.

[26]  Jonah Gabry,et al.  R-squared for Bayesian Regression Models , 2019, The American Statistician.

[27]  Rens van de Schoot,et al.  How the Choice of Distance Measure Influences the Detection of Prior-Data Conflict , 2019, Entropy.

[28]  Michael Evans,et al.  Using Prior Expansions for Prior-Data Conflict Checking , 2019, 1902.10393.

[29]  Aki Vehtari,et al.  Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution , 2017, AISTATS.

[30]  Hajk-Georg Drost,et al.  Philentropy: Information Theory and Distance Quantification with R , 2018, J. Open Source Softw..

[31]  Dan Zhu,et al.  Automated Sensitivity Analysis for Bayesian Inference via Markov Chain Monte Carlo: Applications to Gibbs Sampling , 2018 .

[32]  Georg Heinze,et al.  Variable selection – A review and recommendations for the practicing statistician , 2018, Biometrical journal. Biometrische Zeitschrift.

[33]  Michael I. Jordan,et al.  Covariances, Robustness, and Variational Bayes , 2017, J. Mach. Learn. Res..

[34]  Robin Mitra,et al.  On the Use of Cauchy Prior Distributions for Bayesian Logistic Regression , 2015, Bayesian Analysis.

[35]  Michael Evans,et al.  Optimal Robustness Results for Relative Belief Inferences and the Relationship to Prior-Data Conflict , 2017 .

[36]  Paul-Christian Bürkner,et al.  brms: An R Package for Bayesian Multilevel Models Using Stan , 2017 .

[37]  Andrew Gelman,et al.  The Prior Can Often Only Be Understood in the Context of the Likelihood , 2017, Entropy.

[38]  S. D. Winter,et al.  A Systematic Review of Bayesian Articles in Psychology: The Last 25 Years , 2017, Psychological methods.

[39]  Emanuele Giorgi,et al.  Spatial point patterns:methodology and applications with R , 2017 .

[40]  D. Stoyan Spatial Point Patterns: Methodology and Applications with R. A. Baddeley, E. Rubak, R. Turner (2016). Boca Raton, FL: CRC Press. ISBN: 978‐1‐4822‐1020‐0 (Hardback). , 2017 .

[41]  Virgilio Gómez-Rubio,et al.  Spatial Point Patterns: Methodology and Applications with R , 2016 .

[42]  Luca Scrucca,et al.  mclust 5: Clustering, Classification and Density Estimation Using Gaussian Finite Mixture Models , 2016, R J..

[43]  John Salvatier,et al.  Probabilistic programming in Python using PyMC3 , 2016, PeerJ Comput. Sci..

[44]  J. Gabry,et al.  Bayesian Applied Regression Modeling via Stan , 2016 .

[45]  K. Strimmer,et al.  Optimal Whitening and Decorrelation , 2015, 1512.00809.

[46]  Jilles Vreeken,et al.  Non-parametric Jensen-Shannon Divergence , 2015, ECML/PKDD.

[47]  Sebastian Kurtek,et al.  Bayesian sensitivity analysis with the Fisher–Rao metric , 2015 .

[48]  P. Marriott,et al.  Local and global robustness in conjugate Bayesian analysis , 2015, 1508.07937.

[49]  A. Gelman,et al.  Pareto Smoothed Importance Sampling , 2015, 1507.02646.

[50]  Xiao-Li Meng,et al.  Prior sample size extensions for assessing prior impact and prior‐likelihood discordance , 2014, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[51]  C. Holmes,et al.  Approximate Models and Robust Decisions , 2014, 1402.6118.

[52]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[53]  Leonhard Held,et al.  Sensitivity analysis for Bayesian hierarchical models , 2013, 1312.4797.

[54]  Luca Greco,et al.  A weighted strategy to handle likelihood uncertainty in Bayesian inference , 2013, Comput. Stat..

[55]  Anthony O'Hagan,et al.  Bayesian heavy-tailed models and conflict resolution: A review , 2012 .

[56]  D. Dupuis,et al.  Influence measures and robust estimators of dependence in multivariate extremes , 2011 .

[57]  Hedibert F. Lopes,et al.  Confronting Prior Convictions: On Issues of Prior Sensitivity and Likelihood Robustness in Bayesian Analysis , 2011 .

[58]  Michael Evans,et al.  Weak Informativity and the Information in One Prior Relative to Another , 2011, 1201.1766.

[59]  A. Doucet,et al.  An efficient computational approach for prior sensitivity analysis and cross‐validation , 2010 .

[60]  Patrick Brown,et al.  MCMC for Generalized Linear Mixed Models with glmmBUGS , 2010, R J..

[61]  Gm Gero Walter,et al.  Bayesian Linear Regression — Different Conjugate Models and Their (In)Sensitivity to Prior-Data Conflict , 2010 .

[62]  Hadley Wickham,et al.  ggplot2 - Elegant Graphics for Data Analysis (2nd Edition) , 2017 .

[63]  Ben O'Neill,et al.  Importance sampling for Bayesian sensitivity analysis , 2009, Int. J. Approx. Reason..

[64]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[65]  J. Albert Introduction to Bayesian Thinking , 2009 .

[66]  Laura Ventura,et al.  Robust likelihood functions in Bayesian inference , 2008 .

[67]  Brian Dennis,et al.  Data cloning: easy maximum likelihood estimation for complex ecological models using Bayesian Markov chain Monte Carlo methods. , 2007, Ecology letters.

[68]  P. Diggle,et al.  Model‐based geostatistics , 2007 .

[69]  Sung-Hyuk Cha Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[70]  Michael Evans,et al.  Checking for prior-data conflict , 2006 .

[71]  Carlos J. Perez,et al.  MCMC-based local parametric sensitivity estimations , 2006, Comput. Stat. Data Anal..

[72]  M. A. Best Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[73]  Brian D. Ripley,et al.  Modern Applied Statistics with S Fourth edition , 2002 .

[74]  Andreas Christmann,et al.  Measuring overlap in binary regression , 2001 .

[75]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[76]  Paul Gustafson,et al.  Local Robustness in Bayesian Analysis , 2000 .

[77]  Dale J. Poirier,et al.  REVISING BELIEFS IN NONIDENTIFIED MODELS , 1998, Econometric Theory.

[78]  J. Besag,et al.  Bayesian Computation and Stochastic Systems , 1995 .

[79]  James O. Berger,et al.  An overview of robust Bayesian analysis , 1994 .

[80]  J. Spall,et al.  Sensitivity of a Bayesian Analysis to the Prior Distribution , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[81]  S. Sivaganesan Robust Bayesian diagnostics , 1993 .

[82]  Jianhua Lin,et al.  Divergence measures based on the Shannon entropy , 1991, IEEE Trans. Inf. Theory.

[83]  J. Berger Robust Bayesian analysis : sensitivity to the prior , 1990 .

[84]  D. Hawkins Multivariate Statistics: A Practical Approach , 1990 .

[85]  D. Rubin Using the SIR algorithm to simulate posterior distributions , 1988 .

[86]  J. E. H. Shaw,et al.  Bayesian modelling and sensitivity analysis , 1986 .

[87]  B. Silverman,et al.  Some Aspects of the Spline Smoothing Approach to Non‐Parametric Regression Curve Fitting , 1985 .

[88]  Prem K. Goel,et al.  Information About Hyperparamters in Hierarchical Models , 1981 .

[89]  D. Siegmund Importance Sampling in the Monte Carlo Study of Sequential Tests , 1976 .

[90]  G. Canavos Bayesian estimation - A sensitivity analysis , 1975 .