On-line Mathematics Assessment: The Impact of Mode on Performance and Question Answering Strategies

The transition from paper-based to computer-based assessment raises a number of important issues about how mode might affect children’s performance and question answering strategies. In this project 104 eleven-year-olds were given two sets of matched mathematics questions, one set on-line and the other on paper. Facility values were analyzed to explore the impact of the mode on performance. Errors were coded and this allowed further investigation of the differences between questions in the different modes. The study also investigated children’s affective responses to working on computer, attempting to gain an insight into the effect of motivational factors. This was made possible by observing and interviewing a sub-sample of children. Findings suggested that although there were no statistically significant differences between overall performances on paper and computer, there were enough differences at the individual question-level to warrant further investigation. Close analysis of the data suggests that it is possible that the question type, the way it is asked, and the numbers involved, might interact with mode to affect students’ willingness to show working methods. The findings also suggest that certain types of questions in certain domains might have different impacts according to mode. The study concludes that there is scope for more research to probe further any links that may exist between children’s thinking, behavior and assessment mode in order to satisfy concerns about the relative reliability and validity of computer-based and paper-based testing.