Detecting Misspecified Multilevel Structural Equation Models with Common Fit Indices: A Monte Carlo Study

This study investigated the sensitivity of common fit indices (i.e., RMSEA, CFI, TLI, SRMR-W, and SRMR-B) for detecting misspecified multilevel SEMs. The design factors for the Monte Carlo study were numbers of groups in between-group models (100, 150, and 300), group size (10, 20, 30, and 60), intra-class correlation (low, medium, and high), and the types of model misspecification (Simple and Complex). The simulation results showed that CFI, TLI, and RMSEA could only identify the misspecification in the within-group model. Additionally, CFI, TLI, and RMSEA were more sensitive to misspecification in pattern coefficients while SRMR-W was more sensitive to misspecification in factor covariance. Moreover, TLI outperformed both CFI and RMSEA in terms of the hit rates of detecting the within-group misspecification in factor covariance. On the other hand, SRMR-B was the only fit index sensitive to misspecification in the between-group model and more sensitive to misspecification in factor covariance than misspecification in pattern coefficients. Finally, we found that the influence of ICC on the performance of targeted fit indices was trivial.

[1]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[2]  Michael C Neale,et al.  People are variables too: multilevel structural equations modeling. , 2005, Psychological methods.

[3]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[4]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[5]  K. Wisnicki,et al.  MULTILEVEL FACTOR ANALYSIS OF CLASS AND STUDENT ACHIEVEMENT COMPONENTS CSE Technical Report , 2007 .

[6]  D. Markland.  The golden rule is that there are no golden rules: A commentary on Paul Barrett’s recommendations for reporting model fit in structural equation modelling , 2007 .

[7]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[8]  Kusum Singh,et al.  Book Review: Latent Variable Models: An Introduction to Factor, Path, and Structural Equation Analysis (4th ed.) , 2006 .

[9]  Ehri Ryu,et al.  Level-Specific Evaluation of Model Fit in Multilevel Structural Equation Modeling , 2009 .

[10]  L. Tucker,et al.  A reliability coefficient for maximum likelihood factor analysis , 1973 .

[11]  L. Tucker,et al.  Evaluation of factor analytic research procedures by means of simulated correlation matrices , 1969 .

[12]  Cora J. M. Maas,et al.  The Accuracy of Multilevel Structural Equation Modeling With Pseudobalanced Groups and Small Samples , 2001 .

[13]  H. Marsh,et al.  In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings , 2004 .

[14]  S. West,et al.  Model fit and model selection in structural equation modeling. , 2012 .

[15]  A. Satorra,et al.  Complex Sample Data in Structural Equation Modeling , 1995 .

[16]  J. Neuhaus,et al.  Fixed Effects Regression Methods for Longitudinal Data Using SAS , 2006 .

[17]  J. Hox.  Multilevel analysis: Techniques and applications, 2nd ed. , 2010 .

[18]  Howard T. Everson,et al.  Beyond Individual Differences: Exploring School Effects on SAT Scores , 2004 .

[19]  Satinder P. Singh,et al.  Introduction , 2002, British Journal of Ophthalmology.

[20]  Bengt Muthén,et al.  Multilevel Factor Analysis of Class and Student Achievement Components , 1991 .

[21]  J. Ludden,et al.  Principles and Practice , 1998, Community-based Learning and Social Movements.

[22]  María S. Carlo,et al.  Bilingual phonological awareness: Multilevel construct validation among Spanish-speaking kindergarteners in transitional bilingual education classrooms , 2006 .

[23]  Andrew J. Martin,et al.  Multilevel Motivation and Engagement: Assessing Construct Validity Across Students and Schools , 2010 .

[24]  Rosalie J. Hall,et al.  Applying multilevel confirmatory factor analysis techniques to the study of leadership , 2005 .

[25]  Terry E. Duncan,et al.  Multilevel Covariance Structure Analysis of Sibling Antisocial Behavior. , 1998 .

[26]  Scott L. Thomas,et al.  An introduction to multilevel modeling techniques, 2nd ed. , 2009 .

[27]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[28]  Christine DiStefano,et al.  Using Confirmatory Factor Analysis for Construct Validation: An Empirical Review , 2005 .

[29]  Phil Wood.  Confirmatory Factor Analysis for Applied Research , 2008 .

[30]  Mike W.-L. Cheung,et al.  Applications of Multilevel Structural Equation Modeling to Cross-Cultural Research , 2005 .

[31]  Michael Eid,et al.  Analyzing multitrait-mulitmethod data with multilevel confirmatory factor analysis: An application to the validation of the State-Trait Cheerfulness Inventory , 2011 .

[32]  Ledyard Tucker,et al.  Between, Within, and Total Group Factor Analyses of Student Ratings of Instruction. , 1975, Multivariate behavioral research.

[33]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[34]  Ronald H. Heck,et al.  An Introduction to Multilevel Modeling Techniques , 1999 .

[35]  Kenneth A. Bollen,et al.  An Overview of Structural Equation Models and Recent Extensions , 2007 .

[36]  Oi-Man Kwok,et al.  Using SEM to Analyze Complex Survey Data: A Comparison between Design-Based Single-Level and Model-Based Multilevel Approaches , 2012 .

[37]  B. Muthén,et al.  Multilevel Covariance Structure Analysis , 1994 .

[38]  Kristopher J Preacher,et al.  Reliability estimation in a multilevel confirmatory factor analysis framework. , 2014, Psychological methods.

[39]  Peter M. Bentler,et al.  EQS : structural equations program manual , 1989 .

[40]  Sophia Rabe-Hesketh,et al.  Multilevel Structural Equation Modeling , 2009 .

[41]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[42]  J. Miles,et al.  A time and a place for incremental fit indices , 2007 .

[43]  P. Barrett.  Structural equation modelling : Adjudging model fit , 2007 .

[44]  Ruth Armstrong,et al.  A time and a place , 2013, The Medical journal of Australia.

[45]  Willem E. Saris,et al.  Testing Structural Equation Models or Detection of Misspecifications? , 2009 .

[46]  R. Purc-Stephenson,et al.  Reporting practices in confirmatory factor analysis: an overview and some recommendations. , 2009, Psychological methods.

[47]  G. A. Marcoulides,et al.  New developments and techniques in structural equation modeling , 2001 .

[48]  Cameron N. McIntosh,et al.  Rethinking fit assessment in structural equation modelling: A commentary and elaboration on Barrett (2007) , 2007 .

[49]  R. P. McDonald,et al.  Principles and practice in reporting structural equation analyses. , 2002, Psychological methods.

[50]  Ke-Hai Yuan,et al.  On normal theory based inference for multilevel models with distributional violations , 2002 .

[51]  Ke-Hai Yuan,et al.  3. Multilevel Covariance Structure Analysis by Fitting Multiple Single-Level Models , 2007 .

[52]  Robert C. MacCallum,et al.  Representing sources of error in the common-factor model: Implications for theory and practice. , 1991 .

[53]  James H. Steiger,et al.  Understanding the limitations of global fit assessment in structural equation modeling , 2007 .

[54]  Jiajuan Liang,et al.  An EM algorithm for fitting two-level structural equation models , 2004 .

[55]  Xitao Fan,et al.  Sensitivity of Fit Indexes to Misspecified Structural or Measurement Model Components: Rationale of Two-Index Strategy Revisited , 2005 .