Ordinal test fidelity estimated by an item sampling model

A test theory using only ordinal assumptions is presented. It is based on the idea that the test items are a sample from a universe of items. The sum across items of the ordinal relations for a pair of persons on the universe items is analogous to a true score. Using concepts from ordinal multiple regression, it is possible to estimate the tau correlations of test items with the universe order from the taus among the test items. These in turn permit the estimation of the tau of total score with the universe. It is also possible to estimate the odds that the direction of a given observed score difference is the same as that of the true score difference. The estimates of the correlations between items and universe and between total score and universe are found to agree well with the actual values in both real and artificial data.

[1]  M. Kendall.  Rank Correlation Methods , 1949 .

[2]  Robert J. Mislevy,et al.  Chapter 6: Recent Developments in Item Response Theory with Implications for Teacher Certification , 1987 .

[3]  William Stout,et al.  A New Item Response Theory Modeling Approach with Applications to Unidimensionality Assessment and Ability Estimation , 1990 .

[4]  L. Guttman.  A basis for scaling qualitative data. , 1944 .

[5]  H. Gulliksen.  Theory of mental tests , 1952 .

[6]  J. Loevinger,et al.  The technic of homogeneous tests compared with some aspects of scale analysis and factor analysis. , 1948, Psychological bulletin.

[7]  Frederic M. Lord,et al.  An Analysis of the Verbal Scholastic Aptitude Test Using Birnbaum's Three-Parameter Logistic Model , 1968 .

[8]  F. Samejima.  A New Family of Models for the Multiple-Choice Item. , 1979 .

[9]  Norman Cliff,et al.  Ordinal consistency and ordinal true scores , 1989 .

[10]  Norman Cliff,et al.  An Investigation of Ordinal True Score Test Theory , 1991 .

[11]  H. Wainer,et al.  Some standard errors in item response theory , 1982 .

[12]  Robert H. Somers,et al.  Analysis of Partial Rank Correlation Measures Based on the Product-Moment Model: Part One , 1974 .

[13]  J. Loevinger.  A systematic approach to the construction and evaluation of tests of ability. , 1947 .

[14]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[15]  David Thissen,et al.  A response model for multiple choice items , 1984 .

[16]  Norman Cliff,et al.  Test theory without true scores? , 1979 .

[17]  Frederic M. Lord,et al.  The ‘ability’ scale in item characteristic curve theory , 1975 .

[18]  William Stout,et al.  A nonparametric approach for assessing latent trait unidimensionality , 1987 .

[19]  J. M. Bevan,et al.  Rank Correlation Methods , 1949 .

[20]  Roland K. Hawkes,et al.  The Multivariate Analysis of Ordinal Measures , 1971, American Journal of Sociology.

[21]  Paul Jansen,et al.  A New Derivation of the Rasch Model , 1984 .

[22]  Norman Cliff,et al.  A theory of consistency of ordering generalizable to tailored testing , 1977 .

[23]  N. Cliff,et al.  Variances and Covariances of Kendall's Tau and Their Estimation. , 1991, Multivariate behavioral research.

[24]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[25]  Thomas P. Hettmansperger,et al.  Statistical inference based on ranks , 1985 .

[26]  W. Stout.  A Nonparametric Multidimensional IRT Approach with Applications to Ability Estimation and Test Bias. , 1988 .

[27]  Robert B. Smith.  Continuities in Ordinal Path Analysis , 1974 .

[28]  Charles Lewis,et al.  A Nonparametric Approach to the Analysis of Dichotomous Item Responses , 1982 .

[29]  R. Schulman,et al.  A test theory model for ordinal measurements , 1975 .

[30]  Paul W. Holland,et al.  The Dutch Identity: A New Tool for the Study of Item Response Models. , 1990 .

[31]  T. J. Reynolds,et al.  Assessing the correspondence of one or more vectors to a symmetric matrix using ordinal regression , 1986 .

[32]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[33]  Wendy M. Yen,et al.  THE CHOICE OF SCALE FOR EDUCATIONAL MEASUREMENT: AN IRT PERSPECTIVE , 1986 .

[34]  D. Budescu.  Approximate confidence intervals for a robust scale parameter , 1980 .

[35]  Robert J. Mokken,et al.  A Theory and Procedure of Scale Analysis. , 1973 .

[36]  Robert J. Mislevy,et al.  Recent Developments in Item Response Theory with Implications for Teacher Certification , 1987 .