Item Review and the Rearrangement Procedure: Its process and its results

Permitting item review is to the benefit of the examinees who typically increase their test scores with item review. However, testing companies do not prefer item review since it does not follow the logic on which adaptive tests are based, and since it is prone to cheating strategies. Consequently, item review is not permitted in many adaptive tests. This study attempts to provide a solution that would allow examinees to revise their answers, without jeopardizing the quality and efficiency of the test. The purpose of this study is to test the efficiency of a “rearrangement procedure” that rearranges and skips certain items in order to better estimate the examinees' abilities, without allowing them to cheat on the test. This was examined through a simulation study. The results show that the rearrangement procedure is effective in reducing the standard error of the Bayesian ability estimates and in increasing the reliability of the same estimates.

[1]  Steven L. Wise.  A Critical Analysis of the Arguments for and against Item Review in Computerized Adaptive Testing. , 1996 .

[2]  Steven L. Wise.  Examinee Issues in CAT. , 1997 .

[3]  Mark D. Reckase,et al.  TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS , 1984 .

[4]  Mary Pommerich,et al.  An Examination of Item Review on a CAT Using the Specific Information Item Selection Algorithm , 2001 .

[5]  M Burgess,et al.  Future challenges. , 1982, The New Zealand nursing journal. Kai tiaki.

[6]  Mary E. Lunz,et al.  The Effect of Review on the Psychometric Characteristics of Computerized Adaptive Tests. , 1994 .

[7]  L. Harvill,et al.  Medical students' reasons for changing answers on multiple‐choice tests , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[8]  Howard Wainer.  SOME PRACTICAL CONSIDERATIONS WHEN CONVERTING A LINEARLY ADMINISTERED TEST TO AN ADAPTIVE FORMAT , 1992 .

[9]  Steven L. Wise,et al.  The Relationship between Examinee Anxiety and Preference for Self-Adapted Testing. , 1994 .

[10]  Walter P. Vispoel,et al.  Psychometric Characteristics of Computer-Adaptive and Self-Adaptive Vocabulary Tests: The Role of Answer Feedback and Test Anxiety. , 1998 .

[11]  Steven L. Wise.  Overview of Practical Issues in a CAT Program. , 1997 .

[12]  Mary Pommerich,et al.  From Simulation to Application: Examinees React to Computerized Testing , 2000 .

[13]  Betty A. Bergstrom,et al.  Does Cheating on CAT Pay: NOT! , 1995 .

[14]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[15]  H Pardes,et al.  The future of medical schools and teaching hospitals in the era of managed care , 1997, Academic medicine : journal of the Association of American Medical Colleges.

[16]  C. Douglas Wetzel,et al.  Preliminary psychometric research for CAT-ASVAB: Selecting an adaptive testing strategy. , 1997 .

[18]  Betty A. Bergstrom,et al.  The Effect of Review on Student Ability and Test Efficiency for Computerized Adaptive Tests , 1992 .

[19]  Martha L. Stocking,et al.  PRACTICAL ISSUES IN LARGE‐SCALE HIGH‐STAKES COMPUTERIZED ADAPTIVE TESTING , 1995 .

[20]  H. Wainer.  Computerized adaptive testing: A primer, 2nd ed. , 2000 .

[21]  Walter P. Vispoel,et al.  Reviewing and Changing Answers on Computer‐adaptive and Self‐adaptive Vocabulary Tests , 1998 .

[22]  Walter P. Vispoel,et al.  Individual Differences and Test Administration Procedures: A Comparison of Fixed-Item, Computerized-Adaptive, and Self-Adapted Testing. , 1994 .

[23]  R. Owen,et al.  A Bayesian Sequential Procedure for Quantal Response in the Context of Adaptive Mental Testing , 1975 .

[25]  Donald E. Powers.  TEST ANXIETY AND TEST PERFORMANCE: COMPARING PAPER‐BASED AND COMPUTER‐ADAPTIVE VERSIONS OF THE GRE GENERAL TEST , 1999 .