Did they take the same test? Examinee language proficiency and the structure of language tests

This study investigated the relationship between examinee proficiency and the structure of the Test of English as a Foreign Language (TOEFL) and the Speaking proficiency in English Assessment Kit (SPEAK). Specifically, using multi-group structural equation modeling, this study tested two competing hypotheses about the relationship: whether or not the dimensions of language ability become more or less differentiated as a function of increasing examinee proficiency. Among four alternative models (a second-order model, a correlated specific factor model, a single general factor model, and a totally divisible model), a second-order factor model was established as a baseline model for three ability groups and then estimated simultaneously with cross-group equality imposed. The results showed that the tests were partially equivalent, thereby supporting neither of the hypotheses. Finally, the multiple indicators and multiple causes modeling analysis suggested that the language backgrounds of test-takers were related to measurement non-invariance.

[1]  April Ginther,et al.  Language Background, Ethnicity, and the Internal Construct Validity of the Advanced Placement Spanish Language Examination. , 1995 .

[2]  Bengt Muthen,et al.  Some uses of structural equation modeling in validity studies: Extending IRT to external variables , 1986 .

[3]  Stanley A. Mulaik,et al.  First Order or Higher Order General Factor , 1997 .

[4]  Lyle F. Bachman,et al.  Differential item functioning on two tests of EFL proficiency , 1992 .

[5]  Bengt Muthén,et al.  Simultaneous factor analysis of dichotomous variables in several groups , 1981 .

[6]  David D. Smith,et al.  The Development of Second Language Proficiency , 1992 .

[7]  Brian K. Lynch,et al.  Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants , 1998 .

[8]  Mikyung Kim.  Detecting DIF across the different language groups in a speaking test , 2001 .

[9]  B. Byrne.  Book Review: Structural Equation Modeling with EQS and EQS/Windows: Basic Concepts, Applications, and Programming , 1994 .

[10]  CONFIRMATORY FACTOR ANALYSIS OF THE TEST OF ENGLISH AS A FOREIGN LANGUAGE , 1982 .

[11]  D. Rindskopf,et al.  Some Theory and Applications of Confirmatory Second-Order Factor Analysis. , 1988, Multivariate behavioral research.

[12]  Rick H. Hoyle,et al.  Confirmatory Factor Analysis , 1983 .

[13]  Lyle F. Bachman,et al.  THE CONSTRUCT VALIDATION OF THE FSI ORAL INTERVIEW1 , 1981 .

[14]  An investigation of a criterion- referenced test using G-theory, and factor and cluster analyses , 1992 .

[15]  M. Browne,et al.  Alternative Ways of Assessing Model Fit , 1992 .

[16]  Gary A. Cziko,et al.  The Divisibility of Language Competence: A Confirmatory Approach , 1990 .

[17]  J. Steenkamp,et al.  Assessing Measurement Invariance in Cross-National Consumer Research , 1998 .

[18]  Lyle F. Bachman,et al.  The Construct Validation of Some Components of Communicative Proficiency , 1982 .

[19]  Michael H. Long,et al.  An introduction to second language acquisition research , 1990 .

[20]  James E. Purpura.  Investigating the effects of strategy use and second language test performance with high- and low-ability test takers: a structural equation modelling approach , 1998 .

[21]  N Cliff,et al.  Some Cautions Concerning The Application Of Causal Modeling Methods. , 1983, Multivariate behavioral research.

[22]  James E. Weaver,et al.  Language tests at school , 1979 .

[23]  G. A. Marcoulides,et al.  A First Course in Structural Equation Modeling , 2000 .

[24]  Margaret Thomas,et al.  Assessment of L2 Proficiency in Second Language Acquisition Research , 1994 .

[25]  Lyle F. Bachman,et al.  A Comparison of the Abilities Measured by the Cambridge and Educational Testing Service EFL Test Batteries , 1990 .

[26]  Gordon W. Cheung,et al.  Testing Factorial Invariance across Groups: A Reconceptualization and Proposed New Method , 1999 .

[27]  J. Charles Alderson,et al.  Language Testing in the 1990s: How Far Have We Come? How Much Further Have We to Go?. , 1991 .

[28]  Patrick Allen,et al.  The Development of Second Language Proficiency: The nature of language proficiency , 1990 .

[29]  B. Byrne,et al.  Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. , 1989 .

[30]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[31]  C. Elder.  The Effect of Language Background on “Foreign” Language Test Performance: The Case of Chinese, Italian, and Modern Greek , 1996 .

[32]  B. Wheaton,et al.  Assessment of Fit in Overidentified Models with Latent Variables , 1987 .

[33]  Bernard Spolsky,et al.  An Investigation into the Comparability of Two Tests of English as a Foreign Language , 1995 .

[34]  M. Sasaki.  Second Language Proficiency, Foreign Language Aptitude, and Intelligence: Quantitative and Qualitative Analyses , 1996 .

[35]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[36]  M. Browne,et al.  Cross-Validation Of Covariance Structures. , 1983, Multivariate behavioral research.