Rationale of Computer-Administered Admissible Probability Measurement.

Abstract : The report is part of an ongoing study of decision-theoretic psychometrics. Measurement of a student's knowledge about the subject matter of a multiple-choice question can be much finer if his estimate is elicited, for each possible response, of the probability that it is the correct one. The report describes the rationale underlying a procedure for eliciting such estimates using a proper scoring rule, and new techniques for calibrating those probabilities. The procedure could yield two classes of benefits: students could learn to use the language of numerical probability to communicate uncertainty, and the learning of other subjects could be facilitated. The report also presents new results comparing the incentive for study, rehearsal, and practice provided by the proper scoring rule with that provided by the simple choice procedure, and concerning the potential effect of cutoff scores and prizes on student behavior.