A Third-Order Item Response Theory Model for Modeling the Effects of Domains and Subdomains in Large-Scale Educational Assessment Surveys

Second-order item response theory models have been used for assessments consisting of several domains, such as content areas. We extend the second-order model to a third-order model for assessments that include subdomains nested in domains. Using a graphical model framework, it is shown how the model does not suffer from the curse of multidimensionality. We apply unidimensional, second-order, and third-order item response models to the 2007 Trends in International Mathematics and Science Study. Our findings suggest that deviations from unidimensionality are more pronounced at the content domain level than at the cognitive domain level and that deviations from unidimensionality at the content domain level become negligible after taking into account topic areas.

[1]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[2]  D. Bates,et al.  Approximations to the Log-Likelihood Function in the Nonlinear Mixed-Effects Model , 1995 .

[3]  Jimmy de la Torre,et al.  Simultaneous Estimation of Overall and Domain Abilities: A Higher-Order IRT Model Approach , 2009 .

[4]  K. Jöreskog.  A general method for analysis of covariance structures , 1970 .

[5]  John Schmid,et al.  The development of hierarchical factor solutions , 1957 .

[6]  I. Meilijson.  A fast improvement to the EM algorithm on its own terms , 1989 .

[7]  David Thissen,et al.  On the relationship between the higher-order factor model and the hierarchical factor model , 1999 .

[8]  Karl G. Jöreskog,et al.  On the estimation of polychoric correlations and their asymptotic covariance matrix , 1994 .

[9]  F. Rijmen.  The Latent Class Model as a Measurement Model for Situational Judgment Tests , 2011 .

[10]  Harvey Goldstein,et al.  Improved Approximations for Multilevel Models with Binary Responses , 1996 .

[11]  Robert J. Mislevy,et al.  Recent Developments in the Factor Analysis of Categorical Variables , 1986 .

[12]  S. Lauritzen.  The EM algorithm for graphical association models with missing data , 1995 .

[13]  Frank Rijmen,et al.  Formal Relations and an Empirical Comparison among the Bi‐Factor, the Testlet, and a Second‐Order Multidimensional IRT Model , 2010 .

[14]  Minjeong Jeon,et al.  Modeling Differential Item Functioning Using a Generalization of the Multiple-Group Bifactor Model , 2013 .

[15]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[16]  B. Muthén.  A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators , 1984 .

[17]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm , 1981 .

[18]  Howard Wainer,et al.  Augmented Scores-"Borrowing Strength" to Compute Scores Based on Small Numbers ofltems , 2001 .

[19]  Gordon E. Taub.  A confirmatory analysis of the Wechsler Adult Intelligence Scale-Third edition: Is the verbal/performance discrepancy justified? , 2001 .

[20]  J. Fox,et al.  Bayesian estimation of a multilevel IRT model using gibbs sampling , 2001 .

[21]  P M Bentler,et al.  Multistructure Statistical Model Applied To Factor Analysis. , 1976, Multivariate behavioral research.

[22]  Paul De Boeck,et al.  Latent Class Models for Diary Method Data: Parameter Estimation by Local Computations , 2007, Psychometrika.

[23]  N. Breslow,et al.  Approximate inference in generalized linear mixed models , 1993 .

[24]  L. Thurstone.  Second-order factors , 1944 .

[25]  K. G. Jöreskog,et al.  A GENERAL METHOD FOR ANALYSIS OF COVARIANCE STRUCTURES WITH APPLICATIONS: PART I: GENERAL METHODOLOGY* , 1969 .

[26]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[27]  Francis Tuerlinckx,et al.  A nonlinear mixed model framework for item response theory. , 2003, Psychological methods.

[28]  Donald Hedeker,et al.  Full-information item bi-factor analysis , 1992 .

[29]  Matthias von Davier,et al.  International Large-Scale Assessment Data , 2010 .

[30]  Daniel M. Bolt,et al.  A Comparison of Alternative Models for Testlets , 2006 .

[31]  Matthew S. Johnson,et al.  Marginal Maximum Likelihood Estimation of Item Response Models in R , 2007 .

[32]  S. Sinharay,et al.  An Importance Sampling EM Algorithm for Latent Regression Models , 2007 .

[33]  Shelby J. Haberman,et al.  When Can Subscores Have Value? , 2008 .

[34]  Michael C. Pyryt.  Human cognitive abilities: A survey of factor analytic studies , 1998 .

[35]  Stochastic Approximation Methods for Latent Regression Item Response Models , 2010 .

[36]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[37]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[38]  S. Raudenbush,et al.  Maximum Likelihood for Generalized Linear Models with Nested Random Effects via High-Order, Multivariate Laplace Approximation , 2000 .

[39]  S. Rabe-Hesketh,et al.  Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects , 2005 .

[40]  Shelby J. Haberman,et al.  Reporting of Subscores Using Multidimensional Item Response Theory , 2010 .

[41]  L. Wolfle,et al.  The Three-Stratum Theory of Cognitive Abilities: Test of the Structure of Intelligence across the Life Span. , 1995 .