Fit for a Bayesian: An Evaluation of PPP and DIC for Structural Equation Modeling

Despite its importance to structural equation modeling, model evaluation remains underdeveloped in the Bayesian SEM framework. Posterior predictive p-values (PPP) and deviance information criteria (DIC) are now available in popular software for Bayesian model evaluation, but they remain underutilized. This is largely due to the lack of recommendations for their use. To address this problem, PPP and DIC were evaluated in a series of Monte Carlo simulation studies. The results show that both PPP and DIC are influenced by severity of model misspecification, sample size, model size, and choice of prior. The cutoffs PPP < 0.10 and ∆DIC > 7 work best in the conditions and models tested here to maintain low false detection rates and misspecified model selection rates, respectively. The recommendations provided in this study will help researchers evaluate their models in a Bayesian SEM analysis and set the stage for future development and evaluation of Bayesian SEM fit indices.

[1]  V. Johnson.  UNIFORMLY MOST POWERFUL BAYESIAN TESTS. , 2013, Annals of statistics.

[2]  D. Dey,et al.  To Bayes or Not to Bayes, From Whether to When: Applications of Bayesian Methodology to Modeling , 2004 .

[3]  Aaron M. Ellison,et al.  Bayesian inference in ecology , 2004 .

[4]  Bengt Muthén,et al.  Bayesian structural equation modeling: a more flexible representation of substantive theory. , 2012, Psychological methods.

[5]  Sik-Yum Lee,et al.  Structural equation modelling: A Bayesian approach. , 2007 .

[6]  N. G. Best,et al.  The deviance information criterion: 12 years on , 2014 .

[7]  Cosma Rohilla Shalizi,et al.  Philosophy and the practice of Bayesian statistics. , 2010, The British journal of mathematical and statistical psychology.

[8]  Daniel M. Bolt,et al.  A Comparison of Alternative Models for Testlets , 2006 .

[9]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[10]  Bengt Muthén,et al.  Bayesian Analysis Using Mplus: Technical Implementation , 2010 .

[11]  E. Ward.  A review and comparison of four commonly used Bayesian and maximum likelihood model selection tools , 2008 .

[12]  Zhiyong Zhang,et al.  Bayesian Inference and Application of Robust Growth Curve Models Using Student's t Distribution , 2013 .

[13]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[14]  A. Linde.  DIC in variable selection , 2005 .

[15]  J. Berger.  The case for objective Bayesian analysis , 2006 .

[16]  Sandip Sinharay.  Bayesian item fit analysis for unidimensional item response theory models. , 2006, The British journal of mathematical and statistical psychology.

[17]  R. Schoot,et al.  Bayesian analyses : where to start and what to report , 2014 .

[18]  Surajit Ray,et al.  BIC and Alternative Bayesian Information Criteria in the Selection of Structural Equation Models , 2014, Structural equation modeling : a multidisciplinary journal.

[19]  Sik-Yum Lee,et al.  Basic and Advanced Bayesian Structural Equation Modeling: With Applications in the Medical and Behavioral Sciences , 2012 .

[20]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[21]  A. Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[22]  Xitao Fan,et al.  Sensitivity of Fit Indices to Model Misspecification and Model Types , 2007 .

[23]  Adrian E. Raftery,et al.  Bayesian Model Selection in Structural Equation Models , 1992 .

[24]  D. A. Kenny,et al.  Effect of the Number of Variables on Measures of Fit in Structural Equation Modeling , 2003 .

[25]  Tihomir Asparouhov,et al.  Bayesian Analysis of Latent Variable Models using Mplus , 2010 .

[26]  A. Satorra,et al.  Power of the likelihood ratio test in covariance structure analysis , 1985 .

[27]  M. Jeon,et al.  Decision Qualities of Bayes Factor and p Value-Based Hypothesis Testing , 2017, Psychological methods.

[28]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[29]  A. Linde.  A Bayesian view of model complexity , 2012 .

[30]  M. Plummer.  Penalized loss functions for Bayesian model comparison. , 2008, Biostatistics.

[31]  David B. Dunson,et al.  Bayesian Structural Equation Modeling , 2007 .

[32]  T. Ando.  Predictive Bayesian Model Selection , 2011 .

[33]  R. Scheines,et al.  Bayesian estimation and testing of structural equation models , 1999 .

[34]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[35]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[36]  K. Bollen,et al.  An Empirical Evaluation of the Use of Fixed Cutoff Points in RMSEA Test Statistic in Structural Equation Models , 2008, Sociological methods & research.

[37]  James D. Stamey,et al.  Hidden Dangers of Specifying Noninformative Priors , 2012 .

[38]  A. Linde.  Comment on article by Celeux et al. , 2006 .

[39]  Xiao-Li Meng,et al.  Posterior Predictive $p$-Values , 1994 .

[40]  Clement A. Stone,et al.  Bayesian Comparison of Alternative Graded Response Models for Performance Assessment Applications , 2012 .

[41]  H. Marsh,et al.  In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings , 2004 .

[42]  K. Jöreskog.  A general approach to confirmatory maximum likelihood factor analysis , 1969 .

[43]  Kenneth A. Bollen,et al.  Monte Carlo Experiments: Design and Implementation , 2001 .

[44]  Bengt Muthén,et al.  Bayesian Structural Equation Modeling With Cross-Loadings and Residual Covariances , 2015 .

[45]  J. H. Steiger.  Statistically based tests for the number of common factors , 1980 .

[46]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[47]  Snigdhansu Chatterjee,et al.  Structural Equation Modeling, A Bayesian Approach , 2008, Technometrics.

[48]  N. Hjort,et al.  Post-Processing Posterior Predictive p Values , 2006 .