Anchor-Based Methods for Judgmentally Estimating Item Difficulty Parameters. LSAC Research Report Series.

The purposes of this research study were to develop and field test anchor-based judgmental methods for enabling test specialists to estimate item difficulty statistics. The study consisted of three related field tests. In each, researchers worked with six Law School Admission Test (LSAT) test specialists and one or more of the LSAT subtests. The three field tests produced a number of conclusions. A considerable amount was learned about the process of extracting test specialists' estimates of item difficulty. The ratings took considerably longer to obtain than had been expected. Training, initial ratings, and discussion took a considerable amount of time. Test specialists felt they could be trained to estimate item difficulty accurately and, to some extent, they demonstrated this. Average error in the estimates of item difficulty varied from about 11% to 13 %. Also the discussions were popular with the panelists, and almost always resulted in improved item difficulty estimates. By the end of the study, the two expected frameworks that developers thought they might provide test specialists, had merged to one. Test specialists seemed to benefit from the descriptions of items located at three levels of difficulty and from information about the item statistics of many items. Four appendixes describe tasks and contain the field test materials. (Contains 8 tables and 18 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the ori inal document.

[1]  Robert J. Mislevy,et al.  A TREE‐BASED ANALYSIS OF ITEMS FROM AN ASSESSMENT OF BASIC MATHEMATICS SKILLS , 1994 .

[2]  Roy Freedle,et al.  THE PREDICTION OF TOEFL READING COMPREHENSION ITEM DIFFICULTY FOR EXPOSITORY PROSE PASSAGES FOR THREE ITEM TYPES: MAIN IDEA, INFERENCE, AND SUPPORTING IDEA ITEMS , 1993 .

[3]  R. Mislevy.  Exploiting Auxiliary Information About Items in the Estimation of Rasch Item Difficulty Parameters , 1987 .

[4]  Donald E. Powers,et al.  The Relationship of Content Characteristics of GRE Analytical Reasoning Items to Their Difficulties and Discriminations , 1989 .

[5]  R. Hambleton,et al.  Item Response Theory: Principles and Applications , 1984 .

[6]  Robert J. Mislevy,et al.  Integrating Cognitive and Psychometric Models to Measure Document Literacy. , 1990 .

[7]  Robert J. Mislevy.  Bayes modal estimation in item response models , 1986 .

[8]  H. Swaminathan,et al.  Bayesian Estimation in the Rasch Model , 1982 .

[9]  Kathleen M. Sheehan,et al.  Using Bayesian Decision Theory to Design a Computerized Mastery Test , 1990 .

[10]  Isaac I. Bejar,et al.  Subject Matter Experts' Assessment of Item Statistics , 1981 .

[11]  G. Brown.  On Small-Sample Estimation , 1947 .

[12]  Janice A. Gifford,et al.  Bayesian estimation in the three-parameter logistic model , 1986 .

[13]  H. Swaminathan,et al.  Bayesian estimation in the two-parameter logistic model , 1985 .

[14]  Robert J. Mislevy,et al.  HOW TO EQUATE TESTS WITH LITTLE OR NO DATA , 1992 .