Collapsing factors in multitrait-multimethod models: examining consequences of a mismatch between measurement design and model

Models of confirmatory factor analysis (CFA) are frequently applied to examine the convergent validity of scores obtained from multiple raters or methods in so-called multitrait-multimethod (MTMM) investigations. Many applications of CFA-MTMM and similarly structured models result in solutions in which at least one method (or specific) factor shows non-significant loading or variance estimates. Eid et al. (2008) distinguished between MTMM measurement designs with interchangeable (randomly selected) vs. structurally different (fixed) methods and showed that each type of measurement design implies specific CFA-MTMM measurement models. In the current study, we hypothesized that some of the problems that are commonly seen in applications of CFA-MTMM models may be due to a mismatch between the underlying measurement design and fitted models. Using simulations, we found that models with M method factors (where M is the total number of methods) and unconstrained loadings led to a higher proportion of solutions in which at least one method factor became empirically unstable when these models were fit to data generated from structurally different methods. The simulations also revealed that commonly used model goodness-of-fit criteria frequently failed to identify incorrectly specified CFA-MTMM models. We discuss implications of these findings for other complex CFA models in which similar issues occur, including nested (bifactor) and latent state-trait models.

[1]  Kevin J. Grimm,et al.  Neglect the Structure of Multitrait-Multimethod Data at Your Peril: Implications for Associations With External Variables , 2013 .

[2]  M. Eid,et al.  Structural equation modeling of multitrait-multimethod data: different models for different types of methods. , 2008, Psychological methods.

[3]  R. Steyer,et al.  States and traits in psychological assessment. , 1992 .

[4]  Herbert W. Marsh,et al.  Latent variable models of multitrait-multimethod data. , 1995 .

[5]  Michael Eid,et al.  A CTC(M−1) Model for Different Types of Raters , 2009 .

[6]  A. Richardson,et al.  A Critique of , 2009 .

[7]  M. Eid,et al.  Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model. , 2003, Psychological methods.

[8]  D. Hibbs.  On analyzing the effects of policy interventions : Box-Jenkins and Box-Tiao vs. structural equation models , 1977 .

[9]  Keith F. Widaman,et al.  Multitrait–Multimethod Analysis , 2015 .

[10]  Michael Eid,et al.  Data-Generating Mechanisms Versus Constructively Defined Latent Variables in Multitrait–Multimethod Analysis: A Comment on Castro-Schilo, Widaman, and Grimm (2013) , 2014, Structural equation modeling : a multidisciplinary journal.

[11]  C. Geiser,et al.  Testing for measurement invariance and latent mean differences across methods: interesting incremental information from multitrait-multimethod studies , 2014, Front. Psychol..

[12]  Thomas Götz,et al.  States and Traits , 2017 .

[13]  Michael Eid,et al.  Structural Equation Models for Multitrait-Multimethod Data , 2006 .

[14]  K. Schermelleh-Engel,et al.  Evaluating the Fit of Structural Equation Models: Tests of Significance and Descriptive Goodness-of-Fit Measures. , 2003 .

[15]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[16]  Michael Eid,et al.  Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses , 2015, Behavior research methods.

[17]  D. A. Kenny,et al.  Analysis of the multitrait^multimethod matrix by confirmatory factor analysis , 1992 .

[18]  H. Marsh,et al.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: A Comparison of Alternative Models , 1991 .

[19]  Steven E. Scullen,et al.  A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. , 2002, Psychological methods.

[20]  Valerie F. Reyna,et al.  Random Intercept Item Factor Analysis , 2006 .

[21]  Herbert W. Marsh,et al.  A New , More Powerful Approach to Multitrait-Multimethod Analyses : An Application of Second-order Confirmatory Factor Analysis , 2011 .

[22]  H. Marsh,et al.  A new, more powerful approach to multitrait-multimethod analyses: Application of second-order confirmatory factor analysis. , 1988 .

[23]  M. Dickie.  On The Meaning of ε̉φήμερος , 1976 .

[24]  K. Schermelleh-Engel,et al.  Decomposing person and occasion-specific effects: an extension of latent state-trait (LSI) theory to hierarchical LST models. , 2004, Psychological methods.

[25]  Florian Schmiedek,et al.  Toward an alternative representation for disentangling age-associated differences in general and specific cognitive abilities. , 2004, Psychology and aging.

[26]  J. Laurenceau,et al.  Modeling general and specific variance in multifaceted constructs: a comparison of the bifactor model to other approaches. , 2012, Journal of personality.

[27]  S. West,et al.  A Comparison of Bifactor and Second-Order Models of Quality of Life , 2006, Multivariate behavioral research.

[28]  K. Holzinger,et al.  The Bi-factor method , 1937 .

[29]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[30]  Sophia Rabe-Hesketh,et al.  Structural Equation Models , 2005 .

[31]  Michael Eid,et al.  A multitrait-multimethod model with minimal assumptions , 2000 .

[32]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[33]  S. Reise.  The Rediscovery of Bifactor Measurement Models , 2012 .

[34]  K. Widaman.  Hierarchically Nested Covariance Structure Models for Multitrait-Multimethod Data , 1985 .

[35]  R. Millsap.  The Statistical Analysis of Method Effects in Multitrait-Multimethod Data: A Review , 2014 .

[36]  Michael Eid,et al.  On the meaning of the latent variables in the CT-C(M-1) model: a comment on Maydeu-Olivares and Coffman (2006). , 2008, Psychological methods.