Assessing Dimensionality of a Set of Items - Comparison of Different Approaches

Abstract : This study examines the performance of the following four methodologies for assessing unidimensionality: DIMTEST, Holland and Rosenbaum's approach, linear factor analysis, and nonlinear factor analysis. Each method is examined and compared with other methods on simulated data sets and on real data sets. Seven data sets, all with 2000 examinees, were generated: three unidimensional, and four two-dimensional data sets. Two levels of correlation between abilities were considered: p=.3 and p=.7. Eight different real data sets were used: four of them were expected to be unidimensional, and the other four were expected to be two-dimensional. Findings suggest that, while the linear factor analysis often overestimated the number of underlying dimensions, the other three methods correctly confirmed unidimensionality but differed in their ability to detect lack of unidimensionality. DIMTEST showed excellent power in detecting lack of unidimensionality; Holland and Rosenbaum's and nonlinear factor analysis approaches showed good power, provided the correlation between abilities was low.

[1]  Fritz Drasgow,et al.  Item response theory : application to psychological measurement , 1983 .

[2]  Richard G. Montanelli,et al.  An Investigation of the Parallel Analysis Criterion for Determining the Number of Common Factors , 1975 .

[3]  Lloyd G. Humphreys,et al.  The Primary Mental Ability , 1981 .

[4]  L. Humphreys.  An analysis and evaluation of test and item bias in the prediction context. , 1986 .

[5]  Roderick P. McDonald,et al.  Linear Versus Models in Item Response Theory , 1982 .

[6]  John Hattie,et al.  An Empirical Study of Various Indices for Determining Unidimensionality. , 1984, Multivariate behavioral research.

[7]  W. M. Yen.  Increasing item complexity: A possible cause of scale shrinkage for unidimensional item response theory , 1985 .

[8]  Lloyd G. Humphreys,et al.  Three Approaches to Determining the Dimensionality of Binary Items , 1991 .

[9]  John Hattie,et al.  Methodology Review: Assessing Unidimensionality of Tests and ltenls , 1985 .

[10]  William Stout,et al.  A New Item Response Theory Modeling Approach with Applications to Unidimensionality Assessment and Ability Estimation , 1990 .

[11]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[12]  P. Rosenbaum,et al.  Conditional Association and Unidimensionality in Monotone Latent Variable Models , 1985 .

[13]  Roderick P. McDonald,et al.  The dimensionality of tests and items , 1981 .

[14]  William Stout,et al.  A nonparametric approach for assessing latent trait unidimensionality , 1987 .

[15]  Eugene G. Johnson.  National Assessment of Educational Progress: 1985-86 Public-Use Data Tapes, Version 1.0. Users' Guide. , 1987 .

[16]  Dirk L. Knol,et al.  On the Assessment of Dimensionality in Multidimensional Item Response Theory Models. Research Report 90-8. , 1990 .

[17]  John B. Carroll,et al.  The effect of difficulty and chance success on correlations between items or between tests , 1945 .

[18]  R. P. McDonald,et al.  A second generation nonlinear factor analysis , 1983 .

[19]  Ratna Nandakumar,et al.  Traditional Dimensionality Versus Essential Dimensionality , 1991 .

[20]  M. Reckase.  Unifactor Latent Trait Models Applied to Multifactor Tests: Results and Implications , 1979 .

[21]  Armed Services Vocational Aptitude Battery: Differential Item Functioning on the High School Form. , 1988 .

[22]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[23]  Rebecca Zwick,et al.  Assessing the Dimensionality of NAEP Reading Data , 1987 .

[24]  R. Hambleton,et al.  Assessing the Dimensionality of a Set of Test Items , 1986 .

[25]  C. Parsons,et al.  Application of Unidimensional Item Response Theory Models to Multidimensional Data , 1983 .

[26]  Roderick P. McDonald,et al.  DIFFICULTY FACTORS IN BINARY DATA , 1974 .

[27]  Isaac I. Bejar,et al.  A PROCEDURE FOR INVESTIGATING THE UNIDIMENSIONALITY OF ACHIEVEMENT TESTS BASED ON ITEM PARAMETER ESTIMATES , 1980 .

[28]  R. P. McDonald,et al.  Nonlinear factor analysis. , 1967 .

[29]  Marc E. Gessaroli,et al.  Assessing Test Dimensionality Using an Index Based on Nonlinear Factor Analysis. , 1991 .

[30]  P. Rosenbaum.  Testing the conditional independence and monotonicity assumptions of item response theory , 1984 .

[31]  David A. Harrison,et al.  Robustness of Irt Parameter Estimation to Violations of The Unidimensionality Assumption , 1986 .