I recent years, the need to understand the processes underpinning marking judgements has been highlighted (Vaughan, 1991; Milanovic et al., 1996). This is important, given the continued emphasis on high-stakes examinations in many assessment systems. Improving our understanding of the cognitive processes underlying current assessment systems helps to inform marker training and also leaves us better prepared to anticipate the likely effects of changes or innovations in examining systems. For example, understanding the processes of marking in paper-based marking systems is important in order to anticipate the effects of technological changes such as moves towards on-screen marking. There is a growing research literature on the judgement processes involved in educational assessment but accessing these processes is diffi cult. The most common methodology is verbal protocol analysis involving assessors being asked to ‘think aloud’ whilst marking, with the verbalisations then being analysed. Verbal protocol methodology has come under attack for various reasons but is generally considered a useful method if used appropriately. This article investigates whether thinking aloud affects the marks awarded by examiners and whether markers feel it affects their marking. Whilst this cannot directly confi rm that the processes involved in marking are not affected by thinking aloud, it is likely to provide an indication of the method’s validity. There have been a number of studies about marking processes focused on assessing writing (e.g. Vaughan, 1991; Pula and Huot, 1993; Huot, 1993), sometimes in the context of English as a second language (e.g. Milanovic et al., 1996; Cumming, 1990; Lumley, 2002). In the context of assessments in England, Wales and Northern Ireland, Sanderson (2001) investigated the marking of A level essays in law and sociology, Crisp (2007; in press) investigated the marking of a range of answer types in A level geography, and Greatorex and Suto (2006) researched the marking of maths and business studies GCSE examinations. These studies all included verbal protocol analysis, and between them they have begun to carve out an understanding of marking processes. For example, Milanovic et al. were able to identify a number of broad approaches to the process of marking compositions as well
[1]
Hubert L. Dreyfus,et al.
Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer
,
1987,
IEEE Expert.
[2]
Mike Murphy,et al.
British Educational Research Association Annual Conference
,
2009
.
[3]
J. Greatorex,et al.
What do GCSE examiners think of ‘thinking aloud’? Findings from an exploratory study
,
2008
.
[4]
Timothy D. Wilson,et al.
Telling more than we can know: Verbal reports on mental processes.
,
1977
.
[5]
Alister Cumming,et al.
Expertise in evaluating second language compositions
,
1990
.
[6]
T. Lumley.
Assessment criteria in a large-scale writing test: what do they really mean to the raters?
,
2002
.
[7]
John W. Young,et al.
The Cognitive Equivalence of Reading Comprehension Test Items Via Computerized and Paper-and-Pencil Administration
,
2003
.
[8]
K. A. Ericsson,et al.
Protocol Analysis: Verbal Reports as Data
,
1984
.
[9]
H. Chow.
"Holistic assessment : what goes on in the raters' minds?"
,
1997
.
[10]
Alison Green,et al.
Verbal Protocol Analysis in Language Testing Research: A Handbook
,
1998
.
[11]
H. Dreyfus.
Mind Over Machine
,
1986
.