Bayesian Workflow.

The Bayesian approach to data analysis provides a powerful way to handle uncertainty in all observations, model parameters, and model structure using probability theory. Probabilistic programming languages make it easier to specify and fit Bayesian models, but this still leaves us with many options regarding constructing, evaluating, and using these models, along with many remaining challenges in computation. Using Bayesian inference to solve real-world problems requires not only statistical skills, subject matter knowledge, and programming, but also awareness of the decisions made in the process of data analysis. All of these aspects can be understood as part of a tangled workflow of applied Bayesian statistics. Beyond inference, the workflow also includes iterative model building, model checking, validation and troubleshooting of computational problems, model understanding, and model comparison. We review all these aspects of workflow in the context of several examples, keeping in mind that in practice we will be fitting many models for any given problem, even if only a subset of them will ultimately be relevant for our conclusions.

[1]  M. Ridley Explainable Artificial Intelligence (XAI) , 2022, Information Technology and Libraries.

[2]  J. Riou,et al.  Bayesian workflow for disease transmission modeling in Stan , 2020, Statistics in medicine.

[3]  D. Navarro If Mathematical Psychology Did Not Exist We Might Need to Invent It: A Comment on Theory Building in Psychology , 2020, Perspectives on psychological science : a journal of the Association for Psychological Science.

[4]  Aki Vehtari,et al.  Implicitly adaptive importance sampling , 2019, Statistics and Computing.

[5]  Aki Vehtari,et al.  Adaptive Path Sampling in Metastable Posterior Distributions , 2020, 2009.00471.

[6]  Tuomas Sivula,et al.  Uncertainty in Bayesian Leave-One-Out Cross-Validation Based Model Comparison , 2020, 2008.10296.

[7]  Aki Vehtari,et al.  Regression and Other Stories , 2020 .

[8]  M. Kay ggdist: Visualizations of distributions and uncertainty , 2020 .

[9]  Matthew W. Hoffman,et al.  Black-Box Variational Inference as a Parametric Approximation to Langevin Dynamics , 2020, ICML.

[10]  A. Gelman,et al.  Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors , 2020, J. Mach. Learn. Res..

[11]  S. Bhatt,et al.  Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe , 2020, Nature.

[12]  D. Navarro,et al.  The case for formal methodology in scientific reform , 2020, bioRxiv.

[13]  Aki Vehtari,et al.  Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond , 2020, NeurIPS.

[14]  Andrew Gelman,et al.  Voter Registration Databases and MRP: Toward the Use of Large-Scale Databases in Public Opinion Research , 2020, Political Analysis.

[15]  Aki Vehtari,et al.  $R^*$: A robust MCMC convergence diagnostic with uncertainty using gradient-boosted machines , 2020, 2003.07900.

[16]  Abhraneel Sarma,et al.  Prior Setting in Practice: Strategies and Rationales Used in Choosing Prior Distributions for Bayesian Analysis , 2020, CHI.

[17]  Matthew Kay tidybayes: Tidy Data and Geoms for Bayesian Models , 2020 .

[18]  Bin Yu Veridical data science , 2019, Proceedings of the National Academy of Sciences.

[19]  Michael Evans,et al.  Checking for Prior-Data Conflict Using Prior-to-Posterior Divergences , 2016, Statistical Science.

[20]  H. Bondell,et al.  Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior , 2016, Journal of the American Statistical Association.

[21]  Aki Vehtari,et al.  Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data , 2014, J. Mach. Learn. Res..

[22]  A. Gelman,et al.  Rank-normalization, folding, and localization: An improved R-hat for assessing convergence Rank-Normalization, Folding, and Localization: An Improved (cid:2) R for Assessing Convergence of MCMC An assessing for assessing An improved (cid:2) R for assessing convergence of MCMC , 2020 .

[23]  A. Gelman,et al.  Information, incentives, and goals in election forecasts , 2020, Judgment and Decision Making.

[24]  T. Sterkenburg Deborah G. Mayo: Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars , 2020, Journal for General Philosophy of Science.

[25]  Aki Vehtari,et al.  Making Bayesian Predictive Models Interpretable: A Decision Theoretic Approach , 2019, ArXiv.

[26]  V. Niederlova,et al.  Meta‐analysis of genotype‐phenotype associations in Bardet‐Biedl syndrome uncovers differences among causative genes , 2019, Human mutation.

[27]  Andrew Gelman,et al.  The experiment is just as important as the likelihood in understanding the prior : A cautionary note on robust cognitive modelling , 2019 .

[28]  Pierre Dragicevic,et al.  Increasing the Transparency of Research Papers with Explorable Multiverse Analyses , 2019, CHI.

[29]  Osvaldo A. Martin,et al.  ArviZ a unified library for exploratory analysis of Bayesian models in Python , 2019, J. Open Source Softw..

[30]  Alexander Etz,et al.  Robust Modeling in Cognitive Science , 2019, Computational Brain & Behavior.

[31]  Matthew Kay,et al.  Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths , 2019, CHI.

[32]  Cynthia Rudin,et al.  This Looks Like That: Deep Learning for Interpretable Image Recognition , 2018 .

[33]  Erkan Ozge Buzbas,et al.  Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity , 2018, PloS one.

[34]  Sophia Rabe-Hesketh,et al.  Bayesian Comparison of Latent Variable Models: Conditional Versus Marginal Likelihoods , 2018, Psychometrika.

[35]  A. Gelman,et al.  The garden of forking paths : Why multiple comparisons can be a problem , even when there is no “ fishing expedition ” or “ p-hacking ” and the research hypothesis was posited ahead of time ∗ , 2019 .

[36]  Erik Strumbelj,et al.  Bayesian Combination of Probabilistic Classifiers using Multivariate Normal Mixtures , 2019, J. Mach. Learn. Res..

[37]  Cynthia Rudin,et al.  Please Stop Explaining Black Box Models for High Stakes Decisions , 2018, ArXiv.

[38]  D. Navarro Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection , 2018, Computational Brain & Behavior.

[39]  M. Modrák Reparametrizing the Sigmoid Model of Gene Regulation for Bayesian Inference , 2018, bioRxiv.

[40]  Aki Vehtari,et al.  Validating Bayesian Inference Algorithms with Simulation-Based Calibration , 2018, 1804.06788.

[41]  Sharad Goel,et al.  Disentangling Bias and Variance in Election Polls , 2018 .

[42]  Russell B. Millar,et al.  Conditional vs marginal estimation of the predictive loss of hierarchical models using WAIC and cross-validation , 2018, Stat. Comput..

[43]  Aki Vehtari,et al.  Yes, but Did It Work?: Evaluating Variational Inference , 2018, ICML.

[44]  Michael I. Jordan,et al.  Covariances, Robustness, and Variational Bayes , 2017, J. Mach. Learn. Res..

[45]  Aki Vehtari,et al.  Using Stacking to Average Bayesian Predictive Distributions (with Discussion) , 2017, Bayesian Analysis.

[46]  Aki Vehtari,et al.  Bayesian aggregation of average data: An application in drug development , 2016, The Annals of Applied Statistics.

[47]  H. Rue,et al.  Constructing Priors that Penalize the Complexity of Gaussian Random Fields , 2015, Journal of the American Statistical Association.

[48]  Benjamin Letham,et al.  Forecasting at Scale , 2018, PeerJ Prepr..

[49]  Paul-Christian Bürkner,et al.  brms: An R Package for Bayesian Multilevel Models Using Stan , 2017 .

[50]  Andrew Gelman,et al.  The Prior Can Often Only Be Understood in the Context of the Likelihood , 2017, Entropy.

[51]  Aki Vehtari,et al.  Sparsity information and regularization in the horseshoe and other shrinkage priors , 2017, 1707.01694.

[52]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[53]  Michael Betancourt,et al.  A Conceptual Introduction to Hamiltonian Monte Carlo , 2017, 1701.02434.

[54]  Andrew Gelman,et al.  19 Things We Learned from the 2016 Election , 2017 .

[55]  Lex Nederbragt,et al.  Good enough practices in scientific computing , 2016, PLoS Comput. Biol..

[56]  Dustin Tran,et al.  Automatic Differentiation Variational Inference , 2016, J. Mach. Learn. Res..

[57]  Aki Vehtari,et al.  Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC , 2015, Statistics and Computing.

[58]  Peter Baumgartner,et al.  R – Data Science , 2017 .

[59]  Ivan Rusyn,et al.  A tiered, Bayesian approach to estimating of population variability for regulatory decision-making. , 2017, ALTEX.

[60]  Francis Tuerlinckx,et al.  Increasing Transparency Through a Multiverse Analysis , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[61]  J. Gabry,et al.  Bayesian Applied Regression Modeling via Stan , 2016 .

[62]  Andrea Riebler,et al.  An intuitive Bayesian spatial model for disease mapping that accounts for scaling , 2016, Statistical methods in medical research.

[63]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[64]  Sabine Schulze,et al.  Statistics A Bayesian Perspective , 2016 .

[65]  R. Tibshirani,et al.  Selective Sequential Model Selection , 2015, 1512.02565.

[66]  Joshua R. Loftus Selective inference after cross-validation , 2015, 1511.08866.

[67]  Jaesik Choi,et al.  The Automatic Statistician: A Relational Perspective , 2015, ArXiv.

[68]  Heike Hofmann,et al.  Visualizing statistical models: Removing the blindfold , 2015, Stat. Anal. Data Min..

[69]  A. Gelman,et al.  Pareto Smoothed Importance Sampling , 2015, 1507.02646.

[70]  Jonathan Taylor,et al.  Statistical learning and selective inference , 2015, Proceedings of the National Academy of Sciences.

[71]  Elias Bareinboim,et al.  External Validity: From Do-Calculus to Transportability Across Populations , 2014, Probabilistic and Causal Inference.

[72]  Andrew Gelman,et al.  Diculty of selecting among multilevel models using predictive accuracy , 2015 .

[73]  Kenneth J. Turner,et al.  Workflows for quantitative data analysis in the social sciences , 2015, International Journal on Software Tools for Technology Transfer.

[74]  C. Robert,et al.  Testing hypotheses via a mixture estimation model , 2014, 1412.2044.

[75]  B. Efron Estimation and Accuracy After Model Selection , 2014, Journal of the American Statistical Association.

[76]  Andrew Gelman,et al.  How do we choose our default methods , 2014 .

[77]  Thiago G. Martins,et al.  Penalising Model Component Complexity: A Principled, Practical Approach to Constructing Priors , 2014, 1403.4630.

[78]  Ian M. Mitchell,et al.  Best Practices for Scientific Computing , 2012, PLoS biology.

[79]  M. Betancourt,et al.  Hamiltonian Monte Carlo for Hierarchical Models , 2013, 1312.0906.

[80]  A. Buja,et al.  Valid post-selection inference , 2013, 1306.1059.

[81]  Drew A. Linzer Dynamic Bayesian Forecasting of Presidential Elections in the States , 2013 .

[82]  David B. Dunson,et al.  Bayesian data analysis, third edition , 2013 .

[83]  Leif D. Nelson,et al.  False-Positive Psychology , 2011, Psychological science.

[84]  Elias Bareinboim,et al.  Transportability of Causal and Statistical Relations: A Formal Approach , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[85]  Radford M. Neal MCMC Using Hamiltonian Dynamics , 2011, 1206.1901.

[86]  Jennifer L. Hill,et al.  Bayesian Nonparametric Modeling for Causal Inference , 2011 .

[87]  George Casella,et al.  A Short History of Markov Chain Monte Carlo: Subjective Recollections from Incomplete Data , 2008, 0808.2902.

[88]  Radford M. Neal Probabilistic Inference Using Markov Chain Monte Carlo Methods , 2011 .

[89]  J. Hodges,et al.  Adding Spatially-Correlated Errors Can Mess Up the Fixed Effect You Love , 2010 .

[90]  Jacob M. Montgomery,et al.  Bayesian Model Averaging: Theoretical Developments and Practical Applications , 2010, Political Analysis.

[91]  Christopher D. Manning,et al.  Hierarchical Bayesian Domain Adaptation , 2009, NAACL.

[92]  Joseph Hilbe,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2009 .

[93]  H. Rue,et al.  Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations , 2009 .

[94]  J. Bernardo,et al.  THE FORMAL DEFINITION OF REFERENCE PRIORS , 2009, 0904.0156.

[95]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[96]  Andrew Gelman,et al.  Why We (Usually) Don't Have to Worry About Multiple Comparisons , 2009, 0907.2478.

[97]  J. S. Long,et al.  The Workflow of Data Analysis Using Stata , 2008 .

[98]  Cláudio T. Silva,et al.  Examining Statistics of Workflow Evolution Provenance: A First Study , 2008, SSDBM.

[99]  Xinghua Shi,et al.  SWARM: a scientific workflow for supporting bayesian approaches to improve metabolic models , 2008, CLADE '08.

[100]  Andrew Gelman,et al.  Manipulating and summarizing posterior simulations using random variable objects , 2007, Stat. Comput..

[101]  Christopher Winship,et al.  Counterfactuals and Causal Inference: Methods and Principles for Social Research , 2007 .

[102]  Hal Daumé,et al.  Frustratingly Easy Domain Adaptation , 2007, ACL.

[103]  John Blitzer,et al.  Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification , 2007, ACL.

[104]  S. Martino Approximate Bayesian Inference for Latent Gaussian Models , 2007 .

[105]  Donald B. Rubin,et al.  Validation of Software for Bayesian Models Using Posterior Quantiles , 2006 .

[106]  Hadley Wickham Exploratory model analysis with R and GGobi , 2006 .

[107]  Stan Lipovetsky,et al.  Generalized Latent Variable Modeling: Multilevel,Longitudinal, and Structural Equation Models , 2005, Technometrics.

[108]  Andre G. Journel,et al.  A WORKFLOW FOR MULTIPLE-POINT GEOSTATISTICAL SIMULATION , 2005 .

[109]  Andrew Gelman,et al.  Fully Bayesian Computing , 2004 .

[110]  Steve McConnell,et al.  Code Complete, Second Edition , 2004 .

[111]  A. Gelman Parameterization and Bayesian Modeling , 2004 .

[112]  Chris Volinsky,et al.  Parallel coordinates for exploratory modelling analysis , 2003, Comput. Stat. Data Anal..

[113]  A. Gelman A Bayesian Formulation of Exploratory Data Analysis and Goodness‐of‐fit Testing * , 2003 .

[114]  Andrew Gelman,et al.  Regression Modeling and Meta-Analysis for Decision Making , 2003 .

[115]  Xiao-Li Meng,et al.  The Art of Data Augmentation , 2001 .

[116]  Nando de Freitas,et al.  Sequential Monte Carlo Methods in Practice , 2001, Statistics for Engineering and Information Science.

[117]  C. Mallows Some Comments on Cp , 2000, Technometrics.

[118]  David H. Krantz,et al.  Analysis of Local Decisions Using Hierarchical Modeling, Applied to Home Radon Measurement and Remediation , 1999 .

[119]  John Van Hoewyk,et al.  The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys , 1999 .

[120]  A. Gelman,et al.  Physiological Pharmacokinetic Analysis Using Population Modeling and Informative Prior Distributions , 1996 .

[121]  P N Price,et al.  Bayesian prediction of mean indoor radon concentrations for Minnesota counties. , 1996, Health physics.

[122]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[123]  S Richardson,et al.  A Bayesian approach to measurement error problems in epidemiology using conditional independence models. , 1993, American journal of epidemiology.

[124]  M. Mũgo I will be president , 1992 .

[125]  L. Tierney,et al.  Accurate Approximations for Posterior Moments and Marginal Densities , 1986 .

[126]  J. Jacquez Compartmental analysis in biology and medicine , 1985 .

[127]  D. Rubin Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[128]  M. Degroot,et al.  Information about Hyperparameters in Hierarchical Models , 1981 .

[129]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[130]  M. Stone An Asymptotic Equivalence of Choice of Model by Cross‐Validation and Akaike's Criterion , 1977 .

[131]  M. Stone Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[132]  C. L. Mallows Some comments on C_p , 1973 .

[133]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[134]  Melvin R. Novick,et al.  ESTIMATING MULTIPLE REGRESSIONS IN m GROUPS: A CROSS‐VALIDATION STUDY , 1972 .

[135]  D. Lindley On a Measure of the Information Provided by an Experiment , 1956 .

[136]  W. Deming,et al.  On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known , 1940 .