A Procedure for Dimensionality Analyses of Response Data from Various Test Designs

In some popular test designs (including computerized adaptive testing and multistage testing), many item pairs are not administered to any test takers, which may result in some complications during dimensionality analyses. In this paper, a modified DETECT index is proposed in order to perform dimensionality analyses for response data from such designs. It is proven in this paper that under certain conditions, the modified DETECT can successfully find the dimensionality-based partition of items. Furthermore, the modified DETECT index is decomposed into two parts, which can serve as indices of the reliability of results from the DETECT procedure when response data are judged to be multidimensional. A simulation study shows that the modified DETECT can successfully recover the dimensional structure of response data under reasonable specifications. Finally, the modified DETECT procedure is applied to real response data from two-stage tests to demonstrate how to utilize these indices and interpret their values in dimensionality analyses.

[1]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[2]  Mark D. Reckase,et al.  The Discriminating Power of Items That Measure More Than One Dimension , 1991 .

[3]  Robert J. Mislevy,et al.  Recent Developments in the Factor Analysis of Categorical Variables , 1986 .

[4]  Gregory Camilli,et al.  The Effects of Dimensionality on Equating the Law School Admission Test , 1995 .

[5]  Frederic M. Lord,et al.  A Theoretical Study of Two-Stage Testing. , 1971 .

[6]  William Stout,et al.  The theoretical detect index of dimensionality and its application to approximate simple structure , 1999 .

[7]  R. Darrell Bock,et al.  Feasibility Studies of Two-Stage Testing in Large Scale Educational Assessment: Implications for NAEP. NAEP Validity Studies. Working Paper Series. , 1998 .

[8]  R. Brennan,et al.  Test Equating, Scaling, and Linking: Methods and Practices , 2004 .

[9]  Jinming Zhang,et al.  Conditional Covariance Theory and Detect for Polytomous Items , 2004 .

[10]  William Stout,et al.  A nonparametric approach for assessing latent trait unidimensionality , 1987 .

[11]  Lawrence J. Stricker,et al.  Analyzing test structure by multidimensional scaling , 1990 .

[12]  William Stout,et al.  A New Item Response Theory Modeling Approach with Applications to Unidimensionality Assessment and Ability Estimation , 1990 .

[13]  Fritz Drasgow,et al.  Item response theory : application to psychological measurement , 1983 .

[14]  William A. Sands,et al.  Computerized adaptive testing: From inquiry to operation. , 1997 .

[15]  John Hattie,et al.  Methodology Review: Assessing Unidimensionality of Tests and ltenls , 1985 .

[16]  R. Brennan,et al.  Test Equating, Scaling, and Linking , 2004 .

[17]  Thomas R. Boucher,et al.  Test Equating, Scaling, and Linking: Methods and Practices , 2007 .

[18]  Robert J. Mislevy,et al.  BILOG 3 : item analysis and test scoring with binary logistic models , 1990 .

[19]  P. Holland,et al.  Is It Necessary to Make Anchor Tests Mini-Versions of the Tests Being Equated or Can Some Restrictions Be Relaxed? , 2007 .

[20]  D. Budescu.  EFFICIENCY OF LINEAR EQUATING AS A FUNCTION OF THE LENGTH OF THE ANCHOR TEST , 1985 .

[21]  Rebecca D. Hetter,et al.  Item exposure control in CAT-ASVAB. , 1997 .

[22]  Roderick P. McDonald,et al.  Normal-Ogive Multidimensional Model , 1997 .

[23]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[24]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[25]  Roderick P. McDonald,et al.  The dimensionality of tests and items , 1981 .

[26]  D. Laveault.  Modern theories of measurement : problems and issues , 1994 .

[27]  Hae-Rim Kim.  New techniques for the dimensionality assessment of standardized test data , 1994 .

[28]  John Hattie,et al.  An Empirical Study of Various Indices for Determining Unidimensionality. , 1984, Multivariate behavioral research.

[29]  Andreas Oranje,et al.  Fourth-Grade Students Reading Aloud: NAEP 2002 Special Study of Oral Reading. The Nation's Report Card. NCES 2006-469. , 2005 .

[30]  Klaas Sijtsma,et al.  A Comparative Study of Test Data Dimensionality Assessment Procedures Under Nonparametric IRT Models , 2004 .

[31]  Brian Habing,et al.  Conditional Covariance-Based Nonparametric Multidimensionality Assessment , 1996 .

[32]  L. Roussos,et al.  Formulation of the Detect Population Parameter and Evaluation of Detect Estimator Bias. , 2006 .

[33]  William Stout,et al.  Using New Proximity Measures With Hierarchical Cluster Analysis to Detect Multidimensionality , 1998 .

[34]  W. Hays.  Statistics for the social sciences , 1973 .

[35]  Roderick P. McDonald,et al.  Linear Versus Models in Item Response Theory , 1982 .