The Problem with Having Two Watches: Assessment of Fit When RMSEA and CFI Disagree

ABSTRACT The root mean square error of approximation (RMSEA) and the comparative fit index (CFI) are two widely applied indices to assess fit of structural equation models. Because these two indices are viewed positively by researchers, one might presume that their values would yield comparable qualitative assessments of model fit for any data set. When RMSEA and CFI offer different evaluations of model fit, we argue that researchers are likely to be confused and potentially make incorrect research conclusions. We derive the necessary as well as the sufficient conditions for inconsistent interpretations of these indices. We also study inconsistency in results for RMSEA and CFI at the sample level. Rather than indicating that the model is misspecified in a particular manner or that there are any flaws in the data, the two indices can disagree because (a) they evaluate, by design, the magnitude of the model's fit function value from different perspectives; (b) the cutoff values for these indices are arbitrary; and (c) the meaning of “good” fit and its relationship with fit indices are not well understood. In the context of inconsistent judgments of fit using RMSEA and CFI, we discuss the implications of using cutoff values to evaluate model fit in practice and to design SEM studies.

[1]  Robert Cudeck,et al.  Constructing a covariance matrix that yields a specified minimizer and a specified minimum discrepancy function value , 1992 .

[2]  R. MacCallum,et al.  Power analysis and determination of sample size for covariance structure modeling. , 1996 .

[3]  P. Bentler.  Comparative Fit Indices in Structural Models , 1990 .

[4]  K. Jöreskog,et al.  Analysis of linear structural relationships by maximum likelihood and least squares methods , 1983 .

[5]  D. A. Kenny,et al.  Effect of the Number of Variables on Measures of Fit in Structural Equation Modeling , 2003 .

[6]  Morten Moshagen,et al.  The Model Size Effect in SEM: Inflated Goodness-of-Fit Statistics Are Due to the Size of the Covariance Matrix , 2012 .

[7]  P. Schönemann.  Power as a function of communality in factor analysis , 1981 .

[8]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[9]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[10]  Xitao Fan,et al.  Sensitivity of Fit Indices to Model Misspecification and Model Types , 2007 .

[11]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[12]  K. Yuan,et al.  Robust Procedures in Structural Equation Modeling , 2007 .

[13]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[14]  William N. Venables,et al.  Modern Applied Statistics with S , 2010 .

[15]  Charles E. Heckler,et al.  Applied Multivariate Statistical Analysis , 2005, Technometrics.

[16]  Gregory R. Hancock,et al.  The Reliability Paradox in Assessing Structural Relations Within Covariance Structure Models , 2011 .

[17]  B. Green.  Parameter Sensitivity In Multivariate Methods. , 1977, Multivariate behavioral research.

[18]  J. H. Steiger.  Statistically based tests for the number of common factors , 1980 .

[19]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[20]  Stephen G West,et al.  Evaluating model fit for growth curve models: Integration of fit indices from SEM and MLM frameworks. , 2009, Psychological methods.

[21]  D. Markland.  The golden rule is that there are no golden rules: A commentary on Paul Barrett’s recommendations for reporting model fit in structural equation modelling , 2007 .

[22]  A. Satorra,et al.  Power of the likelihood ratio test in covariance structure analysis , 1985 .

[23]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[24]  Edward E. Rigdon,et al.  The equal correlation baseline model for comparative fit assessment in structural equation modeling , 1998 .

[25]  R. P. McDonald,et al.  Choosing a multivariate model: Noncentrality and goodness of fit. , 1990 .

[26]  Kevin Kim,et al.  The Relation Among Fit Indexes, Power, and Sample Size in Structural Equation Modeling , 2005 .

[27]  R. MacCallum,et al.  Power Analysis in Covariance Structure Modeling Using GFI and AGFI. , 1997, Multivariate behavioral research.

[28]  The equal correlation baseline model: A reply to marsh , 1998 .

[29]  Gary L. Canivez,et al.  Examination of the structural, convergent, and incremental validity of the Reynolds Intellectual Assessment Scales (RIAS) with a clinical sample. , 2012, Psychological assessment.

[30]  George W. Bohrnstedt,et al.  Use of Null Models in Evaluating the Fit of Covariance Structure Models , 1985 .

[31]  Xitao Fan,et al.  Sensitivity of Fit Indexes to Misspecified Structural or Measurement Model Components: Rationale of Two-Index Strategy Revisited , 2005 .

[32]  Laura M. Stapleton,et al.  The Reviewer’s Guide to Quantitative Methods in the Social Sciences , 2018 .

[33]  James H. Steiger,et al.  A note on multiple sample extensions of the RMSEA fit index , 1998 .

[34]  S. Maxwell.  Sample size and multiple regression analysis. , 2000, Psychological methods.

[35]  Edward E. Rigdon,et al.  CFI versus RMSEA: A comparison of two fit indexes for structural equation modeling , 1996 .

[36]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[37]  The equal correlation baseline model: Comment and constructive alternatives , 1998 .

[38]  Ken Kelley,et al.  Accuracy in Parameter Estimation for Targeted Effects in Structural Equation Modeling: Sample Size Planning for Narrow Confidence Intervals , 2022 .

[39]  J. Miles,et al.  A time and a place for incremental fit indices , 2007 .

[40]  P. Dudgeon,et al.  A Note on Extending Steiger's (1998) Multiple Sample RMSEA Adjustment to Other Noncentrality Parameter-Based Statistics , 2004 .

[41]  Ke-Hai Yuan,et al.  On Nonequivalence of Several Procedures of Structural Equation Modeling , 2005 .

[42]  L. Tucker,et al.  A reliability coefficient for maximum likelihood factor analysis , 1973 .

[43]  Ruth Armstrong,et al.  A time and a place , 2013, The Medical journal of Australia.

[44]  Andreas Ritter,et al.  Structural Equations With Latent Variables , 2016 .

[45]  R. P. McDonald,et al.  Principles and practice in reporting structural equation analyses. , 2002, Psychological methods.

[46]  Moritz Heene,et al.  Masking misfit in confirmatory factor analysis by increasing unique variances: a cautionary note on the usefulness of cutoff values of fit indices. , 2011, Psychological methods.

[47]  S. Finney.  Nonnormal and categorical data in structural equation modeling , 2013 .

[48]  Robert W. Elliott,et al.  Reynolds Intellectual Assessment Scales. , 2004 .

[49]  T. Micceri.  The unicorn, the normal curve, and other improbable creatures. , 1989 .

[50]  Kenneth A. Bollen,et al.  Finite Sampling Properties of the Point Estimates and Confidence Intervals of the RMSEA , 2003 .

[51]  R. MacCallum,et al.  Testing differences between nested covariance structure models: Power analysis and null hypotheses. , 2006, Psychological methods.

[52]  Nambury S. Raju,et al.  Accuracy of Population Validity and Cross-Validity Estimation: An Empirical Comparison of Formula-Based, Traditional Empirical, and Equal Weights Procedures , 1999 .

[53]  H. Marsh,et al.  In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings , 2004 .

[54]  Peter M. Bentler,et al.  EQS : structural equations program manual , 1989 .

[55]  Walter Krämer,et al.  Review of Modern applied statistics with S, 4th ed. by W.N. Venables and B.D. Ripley. Springer-Verlag 2002 , 2003 .

[56]  Keith F Widaman,et al.  On specifying the null model for incremental fit indices in structural equation modeling. , 2003, Psychological methods.

[57]  R. Purc-Stephenson,et al.  Reporting practices in confirmatory factor analysis: an overview and some recommendations. , 2009, Psychological methods.

[58]  Roger E. Millsap,et al.  Structural equation modeling made difficult , 2007 .

[59]  R. MacCallum,et al.  When fit indices and residuals are incompatible. , 2002, Psychological methods.

[60]  K. Yuan.  Fit Indices Versus Test Statistics , 2005, Multivariate behavioral research.

[61]  A. Shapiro,et al.  On the multivariate asymptotic distribution of sequential Chi-square statistics , 1985 .