Are fit indices really fit to estimate the number of factors with categorical variables? Some cautionary findings via Monte Carlo simulation.

An early step in the process of construct validation consists of establishing the fit of an unrestricted "exploratory" factorial model for a prespecified number of common factors. For this initial unrestricted model, researchers have often recommended and used fit indices to estimate the number of factors to retain. Despite the logical appeal of this approach, little is known about the actual accuracy of fit indices in the estimation of data dimensionality. The present study aimed to reduce this gap by systematically evaluating the performance of 4 commonly used fit indices-the comparative fit index (CFI), the Tucker-Lewis index (TLI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR)-in the estimation of the number of factors with categorical variables, and comparing it with what is arguably the current golden rule, Horn's (1965) parallel analysis. The results indicate that the CFI and TLI provide nearly identical estimations and are the most accurate fit indices, followed at a step below by the RMSEA, and then by the SRMR, which gives notably poor dimensionality estimates. Difficulties in establishing optimal cutoff values for the fit indices and the general superiority of parallel analysis, however, suggest that applied researchers are better served by complementing their theoretical considerations regarding dimensionality with the estimates provided by the latter method.

[1]  Ulrich Trautwein,et al.  Please Scroll down for Article Structural Equation Modeling: a Multidisciplinary Journal Exploratory Structural Equation Modeling, Integrating Cfa and Efa: Application to Students' Evaluations of University Teaching , 2022 .

[2]  B. Muthén,et al.  How to Use a Monte Carlo Study to Decide on Sample Size and Determine Power , 2002 .

[3]  Wayne F. Velicer,et al.  Construct Explication through Factor or Component Analysis: A Review and Evaluation of Alternative Procedures for Determining the Number of Factors or Components , 2000 .

[4]  K. Bollen,et al.  The Noncentral Chi-square Distribution in Misspecified Structural Equation Models: Finite Sample Results from a Monte Carlo Simulation , 2002, Multivariate behavioral research.

[5]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[6]  Willem E. Saris,et al.  Testing Structural Equation Models or Detection of Misspecifications? , 2009 .

[7]  Sun-Joo Cho,et al.  Accuracy of the Parallel Analysis Procedure With Polychoric Correlations , 2009 .

[8]  M. E. Timmerman,et al.  Using Exploratory Factor Analysis to Determine the Dimensionality of Discrete Responses , 2015 .

[9]  Vicente Ponsoda,et al.  Performance of Velicer’s Minimum Average Partial Factor Retention Method With Categorical Variables , 2011 .

[10]  Sven Reinecke,et al.  The Model-Size Effect on Traditional and Modified Tests of Covariance Structures , 2007 .

[11]  Roger E. Millsap,et al.  Doing the Four-Step Right , 2000 .

[12]  F. Floyd,et al.  Factor analysis in the development and refinement of clinical assessment instruments. , 1995 .

[13]  Pere J. Ferrando,et al.  Unrestricted versus restricted factor analysis of multidimensional test items: some aspects of the problem and some suggestions , 2000 .

[14]  R. MacCallum,et al.  When fit indices and residuals are incompatible. , 2002, Psychological methods.

[15]  H. Kaiser.  The Application of Electronic Computers to Factor Analysis , 1960 .

[16]  Urbano Lorenzo-Seva,et al.  The Hull Method for Selecting the Number of Common Factors , 2011, Multivariate behavioral research.

[17]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[18]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[19]  B. Muthén,et al.  A comparison of some methodologies for the factor analysis of non‐normal Likert variables , 1985 .

[20]  Sanjay Mishra,et al.  Efficient theory development and factor retention criteria: Abandon the ‘eigenvalue greater than one’ criterion , 2008 .

[21]  R. Hoyle,et al.  Latent Variable Models of Need for Uniqueness. , 1996, Multivariate behavioral research.

[22]  Ulf Olsson,et al.  Maximum likelihood estimation of the polychoric correlation coefficient , 1979 .

[23]  K. Jöreskog,et al.  Factor Analysis of Ordinal Variables: A Comparison of Three Approaches , 2001, Multivariate behavioral research.

[24]  B. Muthén,et al.  Exploratory Structural Equation Modeling , 2009 .

[25]  K. Jöreskog,et al.  Confirmatory Factor Analysis of Ordinal Variables With Misspecified Models , 2010 .

[26]  L. Lin,et al.  A concordance correlation coefficient to evaluate reproducibility. , 1989, Biometrics.

[27]  D. Flora,et al.  An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. , 2004, Psychological methods.

[28]  H. Marsh,et al.  In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings , 2004 .

[29]  V. Savalei.  The Relationship Between Root Mean Square Error of Approximation and Model Misspecification in Confirmatory Factor Analysis Models , 2012 .

[30]  H W Marsh,et al.  Is More Ever Too Much? The Number of Indicators per Factor in Confirmatory Factor Analysis. , 1998, Multivariate behavioral research.

[31]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[32]  K. Bollen,et al.  An Empirical Evaluation of the Use of Fixed Cutoff Points in RMSEA Test Statistic in Structural Equation Models , 2008, Sociological methods & research.

[33]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[34]  W. R. Dillon,et al.  A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models , 2005 .

[35]  Francisco José Abad,et al.  A new look at Horn's parallel analysis with ordinal variables. , 2013, Psychological methods.

[36]  Yanyun Yang,et al.  On the number of factors to retain in exploratory factor analysis for ordered categorical data , 2015, Behavior research methods.

[37]  W. Velicer.  Determining the number of components from the matrix of partial correlations , 1976 .

[38]  K. Widaman.  Common Factor Analysis Versus Principal Component Analysis: Differential Bias in Representing Model Parameters? , 1993, Multivariate behavioral research.

[39]  F. Holgado-Tello,et al.  Construct Validity of Likert Scales through Confirmatory Factor Analysis: A Simulation Study Comparing Different Methods of Estimation Based on Pearson and Polychoric Correlations , 2013 .

[40]  Alberto Maydeu-Olivares,et al.  Factor Analysis with Ordinal Indicators: A Monte Carlo Study Comparing DWLS and ULS Estimation , 2009 .

[41]  Pui-Wa Lei,et al.  Evaluating estimation methods for ordinal data in structural equation modeling , 2009 .

[42]  Thomas W. Frazier,et al.  Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? , 2007 .

[43]  L. Taylor,et al.  Examining Listening: Research And Practice In Assessing Second Language Listening , 2013 .

[44]  Nicholas D. Myers.  Coaching Competency and (Exploratory) Structural Equation Modeling: A Substantive-Methodological Synergy. , 2013, Psychology of sport and exercise.

[45]  T. Brown,et al.  Psychometric evaluation of the behavioral inhibition/behavioral activation scales in a large sample of outpatients with anxiety and mood disorders. , 2004, Psychological assessment.

[46]  Steffen Nestler,et al.  A Monte Carlo study comparing PIV, ULS and DWLS in the estimation of dichotomous confirmatory factor analysis. , 2013, The British journal of mathematical and statistical psychology.

[47]  Fritz Drasgow,et al.  Assessing Goodness of Fit: Simple Rules of Thumb Simply Do Not Work , 2011 .

[48]  Jan de Leeuw,et al.  On the relationship between item response theory and factor analysis of discretized variables , 1987 .

[49]  D. A. Kenny,et al.  Effect of the Number of Variables on Measures of Fit in Structural Equation Modeling , 2003 .

[50]  T. Torsheim,et al.  The Strengths and Difficulties Questionnaire in the Bergen Child Study: a conceptually and methodically motivated structural analysis. , 2009, Psychological assessment.

[51]  M. Timmerman,et al.  Dimensionality assessment of ordered polytomous items with parallel analysis. , 2011, Psychological methods.

[52]  Moritz Heene,et al.  Masking misfit in confirmatory factor analysis by increasing unique variances: a cautionary note on the usefulness of cutoff values of fit indices. , 2011, Psychological methods.

[53]  V. Savalei,et al.  When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. , 2012, Psychological methods.

[54]  V. Savalei,et al.  The performance of robust test statistics with categorical data. , 2013, The British journal of mathematical and statistical psychology.

[55]  J. Horn.  A rationale and test for the number of factors in factor analysis , 1965, Psychometrika.

[56]  B. Muthén.  Contributions to factor analysis of dichotomous variables , 1978 .

[57]  Gerhard Mels,et al.  Choosing the Optimal Number of Factors in Exploratory Factor Analysis: A Model Selection Perspective , 2013, Multivariate behavioral research.

[58]  André Beauducel,et al.  On the Performance of Maximum Likelihood Versus Means and Variance Adjusted Weighted Least Squares Estimation in CFA , 2006 .