Effect of Examinee Certainty on Probabilistic Test Scores and a Comparison of Scoring Methods for Probabilistic Responses.

Abstract : This study was an attempt to alleviate some of the difficulties inherent in multiple-choice items by having examinees respond to multiple-choice items in a probabilistic manner. Using this format, examinees are able to respond to each alternative and to provide indications of any partial knowledge they may possess concerning the item. The items used in this study were 30 multiple-choice analogy items. Each item was scored using five different scoring formulas. Three of these scoring formulas were reproducing scoring systems. The fourth scoring method used the probability assigned to the correct alternative as the item score, and the fifth used a function of the absolute difference between the correct response vector for the four alternatives and the actual points assigned to each alternative as the item score. Total test scores for all of the scoring methods were obtained by summing individual item scores. Several studies using probabilistic response methods have shown the effect of a response-style variable, called certainty or risk taking, on scores obtained from probabilistic responses. Results from this study showed a small effect of certainty on the probabilistic scores in terms of the validity of the scores but no effect at all on the factor structure or internal consistency of the scores. The reproducing scoring systems may have an advantage because they maximize examinee's scores when examinees respond honestly, while honest responses will not necessarily maximize an examinee's score with the other two methods. If a reproducing scoring systems is used for this reason, the spherical scoring formula is recommended, since it was the most internally consistent and showed the str ongest first factor of the reproducing scoring systems.