Conflict Diagnostics in Directed Acyclic Graphs, with Applications in Bayesian Evidence Synthesis

Complex stochastic models represented by directed acyclic graphs (DAGs) are increasingly employed to synthesise multiple, imperfect and disparate sources of evidence, to estimate quantities that are difficult to measure directly. The various data sources are dependent on shared parameters and hence have the potential to conflict with each other, as well as with the model. In a Bayesian framework, the model consists of three components: the prior distribution, the assumed form of the likelihood and structural assumptions. Any of these components may be incompatible with the observed data. The detection and quantification of such conflict and of data sources that are inconsistent with each other is therefore a crucial component of the model criticism process. We first review Bayesian model criticism, with a focus on conflict detection, before describing a general diagnostic for detecting and quantifying conflict between the evidence in different partitions of a DAG. The diagnostic is a p-value based on splitting the information contributing to inference about a "separator" node or group of nodes into two independent groups and testing whether the two groups result in the same inference about the separator node(s). We illustrate the method with three comprehensive examples: an evidence synthesis to estimate HIV prevalence; an evidence synthesis to estimate influenza case-severity; and a hierarchical growth model for rat weights.

[1]  H. Moshonov.  Checking for Prior-Data Conßict with Hierarchically SpeciÞed Priors , 2005 .

[2]  David J. Spiegelhalter,et al.  Empirical evaluation of prior beliefs about frequencies : methodology and a case study in congenital heart disease , 1994 .

[3]  Gun Ho Jang,et al.  Invariant P-values for model checking , 2010, 1001.1886.

[4]  Dan Jackson,et al.  Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression‡ , 2012, Research synthesis methods.

[5]  Arthur P. Dempster,et al.  The direct use of likelihood for significance testing , 1997, Stat. Comput..

[6]  N. Hjort,et al.  Post-Processing Posterior Predictive p Values , 2006 .

[7]  G. C. Tiao,et al.  Bayesian inference in statistical analysis , 1973 .

[8]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[9]  Bent Natvig,et al.  Extensions of a Conflict Measure of Inconsistencies in Bayesian Hierarchical Models , 2009 .

[10]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[11]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[12]  Andrew Gelman.  Bayesian Checking of the Second Levels of Hierarchical Models. Comment.. , 2007 .

[13]  David J Spiegelhalter,et al.  Bias modelling in evidence synthesis , 2009, Journal of the Royal Statistical Society. Series A,.

[14]  Anthony O'Hagan,et al.  Bayesian robustness modeling using regularly varying distributions , 2006 .

[15]  S Dias,et al.  Checking consistency in mixed treatment comparison meta‐analysis , 2010, Statistics in medicine.

[16]  Andrew Thomas,et al.  The BUGS project: Evolution, critique and future directions , 2009, Statistics in medicine.

[17]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[18]  R. Kass.  Data-translated likelihood and Jeifreys's rules , 1990 .

[19]  G. Lu,et al.  Assessing Evidence Inconsistency in Mixed Treatment Comparisons , 2006 .

[20]  AE Ades,et al.  Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies‡ , 2012, Research synthesis methods.

[21]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[22]  Nicky J Welton,et al.  Evidence Synthesis in a Decision Modelling Framework , 2012 .

[23]  James Carpenter,et al.  Identifying influential observations in Bayesian models by using Markov chain Monte Carlo , 2012, Statistics in medicine.

[24]  David J. Spiegelhalter,et al.  Use of the false discovery rate when comparing multiple health care providers. , 2008, Journal of clinical epidemiology.

[25]  A Charlett,et al.  Changes in severity of 2009 pandemic A/H1N1 influenza in England: a Bayesian evidence synthesis , 2011, BMJ : British Medical Journal.

[26]  James M. Robins,et al.  Asymptotic Distribution of P Values in Composite Null Models , 2000 .

[27]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[28]  Michael Evans,et al.  Checking for prior-data conflict , 2006 .

[29]  S. Richardson,et al.  Studying place effects on health by synthesising individual and area-level outcomes. , 2008, Social science & medicine.

[30]  David J. Spiegelhalter,et al.  Conflicting evidence in a Bayesian synthesis of surveillance data to estimate human immunodeficiency virus prevalence , 2008 .

[31]  T. Lewis,et al.  Outliers in multilevel data , 1998 .

[32]  S. E. Hills,et al.  Illustration of Bayesian Inference in Normal Data Models Using Gibbs Sampling , 1990 .

[33]  Michael Evans,et al.  Weak Informativity and the Information in One Prior Relative to Another , 2011, 1201.1766.

[34]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[35]  A. P. Dawid,et al.  Present position and potential developments: some personal views , 1984 .

[36]  A E Ades,et al.  Markov Chain Monte Carlo Estimation of a Multiparameter Decision Model: Consistency of Evidence and the Accurate Assessment of Uncertainty , 2002, Medical decision making : an international journal of the Society for Medical Decision Making.

[37]  Paul J. Birrell,et al.  Bayesian modeling to unmask and predict influenza A/H1N1pdm dynamics in London , 2011, Proceedings of the National Academy of Sciences.

[38]  Alex J. Sutton,et al.  Multiparameter evidence synthesis in epidemiology and medical decision‐making: current approaches , 2006 .

[39]  Douglas G. Altman,et al.  Models for potentially biased evidence in meta‐analysis using empirically based priors , 2009 .

[40]  Michael I. Jordan.  Graphical Models , 2003 .

[41]  Jeffrey E. Harris,et al.  Bayes Methods for Combining the Results of Cancer Studies in Humans and other Species , 1983 .

[42]  Nicolas Bousquet,et al.  Diagnostics of prior-data agreement in applied Bayesian analysis , 2008 .

[43]  Michael Evans,et al.  Bayesian ikference procedures derived via the concept of relative surprise , 1997 .

[44]  V. Johnson.  Bayesian Model Assessment Using Pivotal Quantities , 2007 .

[45]  D A Henderson,et al.  Bayesian Calibration of a Stochastic Kinetic Computer Model Using Multiple Data Sources , 2010, Biometrics.

[46]  Sander Greenland,et al.  Relaxation Penalties and Priors for Plausible Modeling of Nonidentified Bias Sources , 2009, 1001.2685.

[47]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[48]  Ida Scheel,et al.  A Graphical Diagnostic for Identifying Influential Model Choices in Bayesian Hierarchical Models , 2010 .

[49]  D. J. Spiegelhalter,et al.  Identifying outliers in Bayesian hierarchical models: a simulation-based approach , 2007 .

[50]  Janneke HilleRisLambers,et al.  High‐dimensional coexistence based on individual variation: a synthesis of evidence , 2010 .

[51]  M. J. Bayarri,et al.  P Values for Composite Null Models , 2000 .

[52]  David J. Spiegelhalter,et al.  Bayesian analysis in expert systems , 1993 .

[53]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[54]  Bent Natvig,et al.  A Robust Conflict Measure of Inconsistencies in Bayesian Hierarchical Models , 2007 .

[55]  David J. Spiegelhalter,et al.  A hierarchical modelling framework for identifying unusual performance in health care providers , 2007 .

[56]  Geir Storvik,et al.  Posterior Predictive p‐values in Bayesian Hierarchical Models , 2009 .

[57]  S. E. Ahmed,et al.  Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference , 2008, Technometrics.

[58]  T. Hothorn,et al.  Simultaneous Inference in General Parametric Models , 2008, Biometrical journal. Biometrische Zeitschrift.

[59]  M. A. Best.  Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .