A Test for Cluster Bias: Detecting Violations of Measurement Invariance Across Clusters in Multilevel Data

We present a test for cluster bias, which can be used to detect violations of measurement invariance across clusters in 2-level data. We show how measurement invariance assumptions across clusters imply measurement invariance across levels in a 2-level factor model. Cluster bias is investigated by testing whether the within-level factor loadings are equal to the between-level factor loadings, and whether the between-level residual variances are zero. The test is illustrated with an example from school research. In a simulation study, we show that the cluster bias test has sufficient power, and the proportions of false positives are close to the chosen levels of significance.

[1]  Barbara M. Byrne,et al.  Testing for Measurement and Structural Equivalence in Large-Scale Cross-Cultural Studies: Addressing the Issue of Nonequivalence , 2010 .

[2]  Carla Rampichini,et al.  Multilevel Factor Models for Ordinal Variables , 2007 .

[3]  Cora J. M. Maas,et al.  The effect of estimation method and sample size in multilevel structural equation modeling , 2010 .

[4]  Keith F Widaman,et al.  Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. , 1993, Psychological bulletin.

[5]  F. Oort.  Using restricted factor analysis to detect item bias , 1992 .

[6]  R E Millsap,et al.  Confirmatory Measurement Model Comparisons Using Latent Means. , 1991, Multivariate behavioral research.

[7]  F. Oort,et al.  Can teachers motivate students to learn? , 2011 .

[8]  Bengt Muthén,et al.  Mean and Covariance Structure Analysis of Hierarchical Data , 2011 .

[9]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[10]  Conor Dolan,et al.  On the likelihood ratio test in structural equation modeling when parameters are subject to boundary constraints. , 2006, Psychological methods.

[11]  Gideon J. Mellenbergh,et al.  Item bias and item response theory , 1989 .

[12]  B. Muthén.  Latent variable modeling in heterogeneous populations , 1989 .

[13]  K. Yuan,et al.  5. Three Likelihood-Based Methods for Mean and Covariance Structure Analysis with Nonnormal Missing Data , 2000 .

[14]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[15]  Christina S. Werner,et al.  Measurement Bias Detection Through Factor Analysis , 2012 .

[16]  D. Sörbom.  A GENERAL METHOD FOR STUDYING DIFFERENCES IN FACTOR MEANS AND FACTOR STRUCTURE BETWEEN GROUPS , 1974 .

[17]  Roger E. Millsap,et al.  Assessing Factorial Invariance in Ordered-Categorical Measures , 2004 .

[18]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[19]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[20]  H. Marsh,et al.  The Factorial Invariance of Student Evaluations of College Teaching , 1984 .

[21]  Jeanne A. Teresi,et al.  An Essay on Measurement and Factorial Invariance , 2006, Medical care.

[22]  Kristopher J Preacher,et al.  A general multilevel SEM framework for assessing multilevel mediation. , 2010, Psychological methods.

[23]  D. Sörbom.  Model modification , 1989 .

[24]  G. J. Mellenbergh,et al.  A Unidimensional Latent Trait Model for Continuous Item Responses. , 1994, Multivariate behavioral research.

[25]  Frans J. Oort,et al.  Using restricted factor analysis with latent moderated structures to detect uniform and nonuniform measurement bias; a simulation study , 2010 .

[26]  G. A. Marcoulides,et al.  Multilevel Analysis Techniques and Applications , 2002 .

[27]  D. Boomsma,et al.  Are intelligence tests measurement invariant over time? Investigating the nature of the Flynn effect , 2004 .

[28]  B. Byrne,et al.  Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. , 1989 .

[29]  Ehri Ryu,et al.  Level-Specific Evaluation of Model Fit in Multilevel Structural Equation Modeling , 2009 .

[30]  Conor V Dolan,et al.  Weak measurement invariance with respect to unmeasured variables: an implication of strict factorial invariance. , 2003, The British journal of mathematical and statistical psychology.

[31]  Bengt Muthén,et al.  On structural equation modeling with data that are not missing completely at random , 1987 .

[32]  Frans J. Oort,et al.  Simulation study of item bias detection with restricted factor analysis , 1998 .

[33]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[34]  H. Goldstein,et al.  Efficient Analysis of Mixed Hierarchical and Cross-Classified Random Structures Using a Multilevel Model , 1994 .

[35]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[36]  Peter M. Bentler,et al.  Ensuring Positiveness of the Scaled Difference Chi-square Test Statistic , 2008, Psychometrika.

[37]  S. Rabe-Hesketh,et al.  Generalized multilevel structural equation modeling , 2004 .

[38]  Noel A. Card,et al.  Latent variable structural equation modeling in cross-cultural research: Multigroup and multilevel approaches , 2008 .

[39]  M. Timmerman,et al.  Cohort differences in Big Five personality factors over a period of 25 years. , 2011, Journal of personality and social psychology.