The Validity of Using Verbal Protocol Analysis to Investigate the Processes Involved in Examination Marking

I recent years, the need to understand the processes underpinning marking judgements has been highlighted (Vaughan, 1991; Milanovic et al., 1996). This is important, given the continued emphasis on high-stakes examinations in many assessment systems. Improving our understanding of the cognitive processes underlying current assessment systems helps to inform marker training and also leaves us better prepared to anticipate the likely effects of changes or innovations in examining systems. For example, understanding the processes of marking in paper-based marking systems is important in order to anticipate the effects of technological changes such as moves towards on-screen marking. There is a growing research literature on the judgement processes involved in educational assessment but accessing these processes is diffi cult. The most common methodology is verbal protocol analysis involving assessors being asked to ‘think aloud’ whilst marking, with the verbalisations then being analysed. Verbal protocol methodology has come under attack for various reasons but is generally considered a useful method if used appropriately. This article investigates whether thinking aloud affects the marks awarded by examiners and whether markers feel it affects their marking. Whilst this cannot directly confi rm that the processes involved in marking are not affected by thinking aloud, it is likely to provide an indication of the method’s validity. There have been a number of studies about marking processes focused on assessing writing (e.g. Vaughan, 1991; Pula and Huot, 1993; Huot, 1993), sometimes in the context of English as a second language (e.g. Milanovic et al., 1996; Cumming, 1990; Lumley, 2002). In the context of assessments in England, Wales and Northern Ireland, Sanderson (2001) investigated the marking of A level essays in law and sociology, Crisp (2007; in press) investigated the marking of a range of answer types in A level geography, and Greatorex and Suto (2006) researched the marking of maths and business studies GCSE examinations. These studies all included verbal protocol analysis, and between them they have begun to carve out an understanding of marking processes. For example, Milanovic et al. were able to identify a number of broad approaches to the process of marking compositions as well