Detecting and Correcting Shift Errors in Standardized Exams

Hundreds of millions multiple choice exams are given every y ear in the United States. Form filling shift errors, where a single absent-minded mismarking displaces a long run of cor re t answers, can substantially reduce scores and invalida te the results of the exam. In this paper, we study the prevalence, impact and detectabi lity of shift errors by analyzing over 100,000 Scholastic Am plitude Tests. We can effectively distinguish approximate ly 50% of the these shifts, and 80% of the substantial shift er rors which reduce the score by 3 or more correct answers. Our resul ts ggest that approximately 1.8% of all SAT exams contain shift errors. Extrapolating these results over all multipl e choice exams implies that students make millions of undete cted shift errors each year. We describe our methods that accurately de tect and correct the majority of significant shift errors whi le guaranteeing nearly no false detections. We argue that our m ethods should be integrated into automated testing procedu res.

[1]  D. N. Harpp,et al.  Crime in the classroom: Detection and prevention of cheating on multiple-choice exams , 1993 .

[2]  John A. Bath,et al.  Answer-Changing Behavior on Objective Examinations , 1967 .

[3]  C. O. Mathews.  Erroneous first impressions on objective tests. , 1929 .

[4]  Leslie J. Briggs,et al.  Should students change their initial answers on objective-type tests? More evidence regarding an old problem. , 1952 .

[5]  Mark Kevin Matter.  The relationship between achievement test response changes ethnicity, and family income , 1985 .

[6]  H. Bradford Thompson.  CHIMPS - A thought-provoker about multiple choice exams , 1983 .

[7]  Pietro J. Pascale,et al.  Changing Initial Answers on Multiple-Choice Achievement Tests. , 1974 .

[8]  Ann Smith.  The Effects of Changing Answers on Scores of Non-Test-Sophisticated Examinees. , 1976 .

[9]  Daniel O. Lynch,et al.  Item Response Changes: Effects on Test Scores. , 1975 .

[10]  Gerald R. Stoffer,et al.  The Consequences of Changing Initial Answers on Objective Tests: A Stable Effect and a Stable Misconception. , 1977 .

[11]  Stanley S. Jacobs,et al.  Answer Changing On Objective Tests: Some Implications for Test Validity , 1972 .

[12]  R. F. Jarrett,et al.  The extra-chance nature of changes in students' responses to objective test-items. , 1948, The Journal of general psychology.

[13]  A. Shwedel,et al.  Some Correlates of Net Gain Resultant from Answer Changing on Objective Achievement Test Items. , 1975 .

[14]  G. E. Hill.  The effect of changed responses in true-false tests. , 1937 .

[15]  Kinnard White,et al.  THE EFFECT OF ITEM TYPE ON THE CONSEQUENCES OF CHANGING ANSWERS ON MULTIPLE CHOICE TESTS , 1979 .

[16]  Michael G. Aamodt,et al.  A Meta-Analytic Investigation of the Effect of Various Test Item Characteristics on Test Scores and Test Completion Times , 1992 .

[17]  David A. Copeland.  Should Chemistry Students Change Answers on Multiple-Choice Tests?. , 1972 .

[18]  E. E. Lamson.  What happens when the second judgment is recorded in a true-false test? , 1935 .

[19]  Eldon Reiling,et al.  A New Approach to the Problem of Changing Initial Responses to Multiple Choice Questions. , 1972 .

[20]  C. C. Crawford,et al.  First impression versus second thought in true-false tests. , 1929 .

[21]  J. Carlson,et al.  Item Sequence and Student Performance on Multiple-Choice Exams: Further Evidence , 1992 .

[22]  K. Anthony Edwards,et al.  First Impressions on Tests: Some New Findings , 1977 .

[23]  Gerardo Prieto,et al.  The Effect of Instructions on Multiple-Choice Test Scores , 1999 .

[24]  Michael D. Beck.  The Effect of Item Response Changes on Scores on an Elementary Reading Achievement Test , 1978 .

[25]  Melvin M. Mark,et al.  Affective Cues and Processing Strategy: Color-Coded Examination Forms Influence Performance , 1998 .

[26]  W. Balch,et al.  Item Order Affects Performance on Multiple-Choice Exams , 1989 .

[27]  Steven Skiena,et al.  Shift error detection in standardized exams , 2000, J. Discrete Algorithms.