Exploring the Robustness of a Unidimensional Item Response Theory Model With Empirically Multidimensional Data

ABSTRACT Unidimensionality and local independence are two common assumptions of item response theory. The former implies that all items measure a common latent trait, while the latter implies that responses are independent, conditional on respondents’ location on the latent trait. Yet, few tests are truly unidimensional. Unmodeled dimensions may result in test items displaying dependencies, which can lead to misestimated parameters and inflated reliability estimates. In this article, we investigate the dimensionality of interim mathematics tests and evaluate the extent to which modeling minor dimensions in the data change model parameter estimates. We found evidence of minor dimensions, but parameter estimates across models were similar. Our results indicate that minor dimensions outside the primary trait have negligible consequences on parameter estimates. This finding was observed despite the ratio of multidimensional to unidimensional items being above previously recommended thresholds.

[1]  J. Horn.  A rationale and test for the number of factors in factor analysis , 1965, Psychometrika.

[2]  Christine E. DeMars.  Application of the Bi-Factor Multidimensional Item Response Theory Model to Testlet-Based Tests. , 2006 .

[3]  Daniel M. Bolt,et al.  Estimation of Compensatory and Noncompensatory Multidimensional Item Response Models Using Markov Chain Monte Carlo , 2003 .

[4]  B. Muthén.  Contributions to factor analysis of dichotomous variables , 1978 .

[5]  Akihito Kamata,et al.  A Note on the Relation Between Factor Analytic and Item Response Theory Models , 2008 .

[6]  Yeh-Tai Chou,et al.  Checking Dimensionality in Item Response Models With Principal Component Analysis on Standardized Residuals , 2010 .

[7]  W Revelle,et al.  Very Simple Structure: An Alternative Procedure For Estimating The Optimal Number Of Interpretable Factors. , 1979, Multivariate behavioral research.

[8]  Nilufer Kahraman.  Unidimensional Interpretations for Multidimensional Test Items , 2013 .

[9]  F. Holgado-Tello,et al.  Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables , 2008 .

[10]  Stephen G. Sireci,et al.  ON THE RELIABILITY OF TESTLET‐BASED TESTS , 1991 .

[11]  Edward Haksing Ip,et al.  Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models. , 2010, The British journal of mathematical and statistical psychology.

[12]  W. Revelle.  psych: Procedures for Personality and Psychological Research , 2017 .

[13]  April L. Zenisky,et al.  Identification and Evaluation of Local Item Dependencies in the Medical College Admissions Test , 2002 .

[14]  Christine E. DeMars.  A Tutorial on Interpreting Bifactor Model Scores , 2013 .

[15]  Frank Rijmen,et al.  Formal Relations and an Empirical Comparison among the Bi‐Factor, the Testlet, and a Second‐Order Multidimensional IRT Model , 2010 .

[16]  Hadley Wickham,et al.  ggplot2 - Elegant Graphics for Data Analysis (2nd Edition) , 2017 .

[17]  David R. Anderson,et al.  Multimodel Inference , 2004 .

[18]  W. Velicer,et al.  Comparison of five rules for determining the number of components to retain. , 1986 .

[19]  Francisco Pablo Holgado Tello,et al.  Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables , 2010 .

[20]  J. Horn,et al.  Cattell's Scree Test In Relation To Bartlett's Chi-Square Test And Other Observations On The Number Of Factors Problem. , 1979, Multivariate behavioral research.

[21]  Chong Ho Yu,et al.  A data visualization and data mining approach to response and non-response analysis in survey research , 2007 .

[22]  R. Henson,et al.  Use of Exploratory Factor Analysis in Published Research , 2006 .

[23]  William Stout,et al.  A nonparametric approach for assessing latent trait unidimensionality , 1987 .

[24]  C. Parsons,et al.  Application of Unidimensional Item Response Theory Models to Multidimensional Data , 1983 .

[25]  R. Lissitz,et al.  Applying Multidimensional Item Response Theory Models in Validating Test Dimensionality: An Example of K-12 Large-Scale Science Assessment. , 2012 .

[26]  Anders Christoffersson,et al.  Factor analysis of dichotomized variables , 1975 .

[27]  William Revelle,et al.  An overview of the psych package , 2009 .

[28]  Brenda Sugrue,et al.  The Lack of Fidelity between Cognitively Complex Constructs and Conventional Test Development Practice. , 2005 .

[29]  Wendy M. Yen,et al.  Scaling Performance Assessments: Strategies for Managing Local Item Dependence , 1993 .

[30]  Rianne Janssen,et al.  Modeling Item-Position Effects Within an IRT Framework , 2012 .

[31]  Howard Wainer,et al.  Precision and Differential Item Functioning on a Testlet-Based Test: The 1991 Law School Admissions Test as an Example , 1995 .

[32]  Ratna Nandakumar,et al.  Traditional Dimensionality Versus Essential Dimensionality , 1991 .

[33]  William F. Strout.  A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation , 1990 .

[34]  F. Drasgow,et al.  Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. , 1983 .

[35]  David A. Harrison,et al.  Robustness of Irt Parameter Estimation to Violations of The Unidimensionality Assumption , 1986 .

[36]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[37]  W. Velicer.  Determining the number of components from the matrix of partial correlations , 1976 .

[38]  Pedro M. Valero-Mora,et al.  Determining the Number of Factors to Retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis , 2007 .

[39]  Sanjay Mishra,et al.  Efficient theory development and factor retention criteria: Abandon the ‘eigenvalue greater than one’ criterion , 2008 .