Modeling the multidimensional structure of students' foreign language competence within and between classrooms

Combining multilevel (ML) analysis and multidimensional item response theory (MIRT) provides a valuable method for analyzing data of educational assessments, where clustered data (e.g., students in classes) and multidimensional constructs frequently occur. It allows to model multiple ability dimensions while simultaneously taking the hierarchical structure into account. The dimensional structure of students’ foreign language competence within and between classrooms was investigated by applying a ML-MIRT measurement model to data of N = 9,410 students in 427 classes who had answered three different subtests of English as a foreign language. Results were compared to a MIRT model not taking into account the multilevel structure. A markedly more differentiated correlation structure is found within classrooms compared with the betweenclassroom level and compared with the model without multilevel structure. Results show that by modeling the latent multilevel structure, estimation and interpretation of ability profiles can be possible even with highly correlated ability dimensions.

[1]  Mark D. Reckase,et al.  A Linear Logistic Multidimensional Model for Dichotomous Item Response Data , 1997 .

[2]  L. Corrado.  Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models , 2005 .

[3]  Jean-Paul Fox,et al.  Applications of Multilevel IRT Modeling , 2004 .

[4]  Akihito Kamata,et al.  Multilevel Rasch Models , 2007 .

[5]  Michael C Neale,et al.  People are variables too: multilevel structural equations modeling. , 2005, Psychological methods.

[6]  Lihua Yao,et al.  A Multidimensional Item Response Modeling Approach for Improving Subscale Proficiency Estimation and Classification , 2007 .

[7]  T. A. Warm.  Weighted likelihood estimation of ability in item response theory , 1989 .

[8]  Bengt Muthén,et al.  Multilevel Factor Analysis of Class and Student Achievement Components , 1991 .

[9]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[10]  Jean-Paul Fox,et al.  Multilevel IRT model assessment , 2005 .

[11]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[12]  Andreas Schleicher,et al.  PISA 2006: Science Competencies for Tomorrow's World , 2007 .

[13]  G. A. Marcoulides.  Multilevel Analysis Techniques and Applications , 2002 .

[14]  Y. Poortinga,et al.  Structural Equivalence in Multilevel Research , 2002 .

[15]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[16]  Jan de Leeuw,et al.  Introducing Multilevel Modeling , 1998 .

[17]  Wen-Chung Wang,et al.  Improving measurement precision of test batteries using multidimensional item response models. , 2004, Psychological methods.

[18]  Akihito Kamata,et al.  Item Analysis by the Hierarchical Generalized Linear Model. , 2001 .

[19]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[20]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[21]  Wen-Chung Wang,et al.  Multidimensional Rasch Analysis of a Psychological Test With Multiple Subtests , 2009 .

[22]  Jean-Paul Fox,et al.  Bayesian modeling of measurement error in predictor variables using item response theory , 2003 .

[23]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[24]  J. Gill.  Hierarchical Linear Models , 2005 .

[25]  Margaret Wu,et al.  ACER conquest: generalised item response modelling software , 1998 .

[26]  Dena A. Pastor,et al.  The Use of Multilevel Item Response Theory Modeling in Applied Research: An Illustration , 2003 .

[27]  Ann A. O'Connell,et al.  Multilevel modeling of educational data , 2008 .

[28]  Representation of Competencies in Multidimensional IRT Models with Within-Item and Between-Item Multidimensionality , 2008 .

[29]  J-P Fox,et al.  Multilevel IRT using dichotomous and polytomous response data. , 2005, The British journal of mathematical and statistical psychology.

[30]  A. Alas,et al.  BAYESIAN ESTIMATION OF A MULTILEVEL IRT MODEL USING GIBBS SAMPLING JEAN-PAUL FOX AND CEES , 2005 .

[31]  Irene-Anna N. Diakidoy,et al.  THE RELATIONSHIP BETWEEN LISTENING AND READING COMPREHENSION OF DIFFERENT TYPES OF TEXT AT INCREASING GRADE LEVELS , 2005 .

[32]  B. Muthén,et al.  The multilevel latent covariate model: a new, more reliable approach to group-level effects in contextual studies. , 2008, Psychological methods.

[33]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[34]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[35]  J. Fox,et al.  Bayesian estimation of a multilevel IRT model using gibbs sampling , 2001 .

[36]  André A. Rupp,et al.  An NCME Instructional Module on Booklet Designs in Large‐Scale Assessments of Student Achievement: Theory and Practice , 2009 .

[37]  Eckhard Klieme,et al.  Unterricht und Kompetenzerwerb in Deutsch und Englisch. Ergebnisse der DESI-Studie , 2008 .

[38]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[39]  Michael C. Pyryt.  Human cognitive abilities: A survey of factor analytic studies , 1998 .

[40]  Stephen W. Raudenbush,et al.  A Multilevel, Multivariate Model for Studying School Climate With Estimation Via the EM Algorithm and Application to U.S. High-School Data , 1991 .

[41]  Albert Satorra,et al.  A scaled difference chi-square test statistic for moment structure analysis , 1999 .

[42]  B. Muthén,et al.  Multilevel Covariance Structure Analysis , 1994 .