Measurement Decision Theory.

This paper describes and evaluates a decision theory measurement model that can be used to classify examinees based on their item response patterns. The model has a simple framework that starts with the conditional probabilities of examinees in each category or mastery state responding correctly to each item. An overview of measurement decision theory and its key concepts are presented and illustrated using a binary classification (pass/fail) test and a sample three-item test. The research presents an evaluation of the model by examining the: (1) classification accuracy of tests scored using measurement decision theory; (2) differential sequential testing procedures by comparing classification accuracy against that of the best case item response theory scenario; (3) the number of items needed to make a classification; and (4) the number of examinees needed to calibrate measurement decision theory item parameters satisfactorily. The research shows that a large percentage of examinees can be classified accurately with very few items and that surprisingly few examinees are needed for calibration.

[1]  Chuan-Ju Lin,et al.  Effects of Item-Selection Criteria on Classification Testing with the Sequential Probability Ratio Test. ACT Research Report Series. , 2000 .

[2]  C. Mitchell Dayton,et al.  The Use of Probabilistic Models in the Assessment of Mastery , 1977 .

[3]  Theodore W. Frick,et al.  Computerized adaptive testing in instructional settings , 1993 .

[4]  Mark D. Reckase,et al.  Comparison of SPRT and Sequential Bayes Procedures for Classifying Examinees Into Two Categories Using a Computerized Test , 1996 .

[5]  Kathleen M. Sheehan,et al.  Using Bayesian Decision Theory to Design a Computerized Mastery Test , 1990 .

[6]  Frank B. Baker,et al.  The Basics of Item Response Theory. Second Edition. , 2001 .

[7]  David J. Weiss,et al.  Book Review : New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing David J. Weiss (Ed.) New York: Academic Press, 1983, 345 pp., $35.00 , 1984 .

[8]  James E. Carlson,et al.  The NAEP 1996 Technical Report. , 1999 .

[9]  Willem J. van der Linden,et al.  A compensatory approach to optimal selection with mastery scores , 1994 .

[10]  Wim J. van der Linden,et al.  Coefficients for Tests from a Decision Theoretic Point of View , 1977 .

[11]  David J. Weiss,et al.  A Comparison of IRT-Based Adaptive Mastery Testing and a Sequential Mastery Testing Procedure , 1983 .

[12]  F. Baker.  The basics of item response theory , 1985 .

[13]  L. Cronbach,et al.  Psychological tests and personnel decisions , 1958 .

[14]  Kathleen M. Sheehan,et al.  Computerized Mastery Testing With Nonequivalent Testlets , 1990 .

[15]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[16]  Mark D. Reckase,et al.  A Procedure for Decision Making Using Tailored Testing , 1983 .

[17]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[18]  J. Andel.  Sequential Analysis , 2022, The SAGE Encyclopedia of Research Design.

[19]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[20]  Mark D. Reckase,et al.  The Selection of Test Items for Decision Making with a Computer Adaptive Test. , 1994 .

[21]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[22]  Ronald K. Hambleton,et al.  TOWARD AN INTEGRATION OF THEORY AND METHOD FOR CRITERION-REFERENCED TESTS1,2 , 1973 .

[23]  R. Wood.  Adaptive Testing: A Bayesian Procedure for the Efficient Measurement of Ability. , 1976 .

[24]  Theodorus Johannes Hendrikus Maria Eggen,et al.  Item Selection in Adaptive Testing with the Sequential Probability Ratio Test , 1999 .

[25]  George B. Macready,et al.  The application of latent class models in adaptive testing , 1992 .