It Matters: Reference Indicator Selection in Measurement Invariance Tests

Conventional approaches for selecting a reference indicator (RI) could lead to misleading results in testing for measurement invariance (MI). Several newer quantitative methods have been available for more rigorous RI selection. However, it is still unknown how well these methods perform in terms of correctly identifying a truly invariant item to be an RI. Thus, Study 1 was designed to address this issue in various conditions using simulated data. As a follow-up, Study 2 further investigated the advantages/disadvantages of using RI-based approaches for MI testing in comparison with non-RI-based approaches. Altogether, the two studies provided a solid examination on how RI matters in MI tests. In addition, a large sample of real-world data was used to empirically compare the uses of the RI selection methods as well as the RI-based and non-RI-based approaches for MI testing. In the end, we offered a discussion on all these methods, followed by suggestions and recommendations for applied researchers.

[1]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[2]  Bengt Muthén,et al.  Bayesian structural equation modeling: a more flexible representation of substantive theory. , 2012, Psychological methods.

[3]  Holger Steinmetz,et al.  Estimation and Comparison of Latent Means Across Cultures , 2018 .

[4]  Albert Maydeu-Olivares,et al.  Understanding the Model Size Effect on SEM Fit Indices , 2018, Educational and psychological measurement.

[5]  Myeongsun Yoon,et al.  Comparisons of Three Empirical Methods for Partial Factorial Invariance: Forward, Backward, and Factor-Ratio Tests , 2016 .

[6]  B. Chorpita,et al.  Assessment of symptoms of DSM-IV anxiety and depression in children: a revised child anxiety and depression scale. , 2000, Behaviour research and therapy.

[7]  J. Horn,et al.  When is invariance not invarient: A practical scientist's look at the ethereal concept of factor invariance. , 1983 .

[8]  A. Meade.  A taxonomy of effect size measures for the differential functioning of items and scales. , 2010, The Journal of applied psychology.

[9]  Myeongsun Yoon,et al.  Testing Measurement Invariance Using MIMIC , 2012 .

[10]  Fritz Drasgow,et al.  Detecting differential item functioning with confirmatory factor analysis and item response theory: toward a unified strategy. , 2006, The Journal of applied psychology.

[11]  Gordon W. Cheung,et al.  Testing Measurement Models for Factorial Invariance: A Systematic Approach , 1998 .

[12]  Detecting Differential Item Functioning , 2015 .

[13]  J. Steenkamp,et al.  Assessing Measurement Invariance in Cross-National Consumer Research , 1998 .

[14]  Roger E. Millsap,et al.  Detecting Violations of Factorial Invariance Using Data-Based Specification Searches: A Monte Carlo Study , 2007 .

[15]  R. Millsap,et al.  Evaluating the impact of partial factorial invariance on selection in two populations. , 2004, Psychological methods.

[16]  Dexin Shi,et al.  The Impact of Partial Factorial Invariance on Cross-Group Comparisons , 2019, Assessment.

[17]  Larry Wasserman,et al.  All of Statistics: A Concise Course in Statistical Inference , 2004 .

[18]  E. S. Kim,et al.  Testing Measurement Invariance: A Comparison of Multiple-Group Categorical CFA and IRT , 2011 .

[19]  S. Reise,et al.  Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. , 1997 .

[20]  Robert D. Ankenmann,et al.  An Investigation of the Power of the Likelihood Ratio Goodness-of-Fit Statistic in Detecting Differential Item Functioning. , 1999 .

[21]  R. Terry,et al.  Bayesian SEM for Specification Search Problems in Testing Factorial Invariance , 2017, Multivariate behavioral research.

[22]  Natalie A. Wright,et al.  Solving the measurement invariance anchor item problem in item response theory. , 2012, The Journal of applied psychology.

[23]  B. Byrne,et al.  Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. , 1989 .

[24]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[25]  Daniel McNeish,et al.  The Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models , 2018, Journal of personality assessment.

[26]  R. Vandenberg,et al.  A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research , 2000 .

[27]  Tenko Raykov,et al.  On the Dependability of a Popular Procedure for Studying Measurement Invariance: A Cause for Concern? , 2020, Structural Equation Modeling: A Multidisciplinary Journal.

[28]  Emily C. Johnson,et al.  The Role of Referent Indicators in Tests of Measurement Invariance , 2009 .

[29]  Carol M. Woods.  Empirical Selection of Anchors for Tests of Differential Item Functioning , 2009 .

[30]  K. G. J8reskoC,et al.  Simultaneous Factor Analysis in Several Populations , 2007 .

[31]  Moritz Heene,et al.  Masking misfit in confirmatory factor analysis by increasing unique variances: a cautionary note on the usefulness of cutoff values of fit indices. , 2011, Psychological methods.

[32]  B. French,et al.  Multigroup Confirmatory Factor Analysis: Locating the Invariant Referent Sets , 2008 .

[33]  Gregory R. Hancock,et al.  The Reliability Paradox in Assessing Structural Relations Within Covariance Structure Models , 2011 .

[34]  J. Horn,et al.  A practical and theoretical guide to measurement invariance in aging research. , 1992, Experimental aging research.

[35]  Adam W. Meade,et al.  A Monte-Carlo Study of Confirmatory Factor Analytic Tests of Measurement Equivalence/Invariance , 2004 .

[36]  O. Chernyshenko,et al.  The Effects of Referent Item Parameters on Differential Item Functioning Detection Using the Free Baseline Likelihood Ratio Test , 2009 .

[37]  Albert Maydeu-Olivares,et al.  Evaluating Factorial Invariance: An Interval Estimation Approach Using Bayesian Structural Equation Modeling , 2019, Multivariate behavioral research.

[38]  Larry Wasserman,et al.  All of Statistics , 2004 .