Assessing paired orals: Raters' orientation to interaction

Speaking tasks involving peer-to-peer candidate interaction are increasingly being incorporated into language proficiency assessments, in both large-scale international testing contexts, and in smaller-scale, for example course-related, ones. This growth in the popularity and use of paired and group orals has stimulated research, particularly into the types of discourse produced and the possible impact of candidate background factors on performance. However, despite the fact that the strongest argument for the validity of peer-to-peer assessment lies in the claim that such tasks allow for the assessment of a broader range of interactional skills than the more traditional interview-format tests do, there is surprisingly little research into the judgments that are made of such performances. The fact that raters, and rating criteria, are in a crucial mediating position between output and outcomes, warrants investigation into how raters construe the interaction in these tasks. Such investigations have the potential to inform the development of interaction-based rating scales and ensure that validity claims are moved beyond the content level to the construct level. This paper reports the findings of a verbal protocol study of teacher-raters viewing the paired test discourse of 17 beginner dyads in a university-based Spanish as a foreign language course. The findings indicate that the raters identified three interaction parameters: non-verbal interpersonal communication, interactive listening, and interactional management. The findings have implications for our understanding of the construct of effective interaction in paired candidate speaking tests, and for the development of appropriate rating scales.

[1]  Susan M. Gass,et al.  Input in second language acquisition , 1985 .

[2]  K. Bailey.  The Research Manual: Design and Statistics for Applied Linguistics , 1991 .

[3]  Rod Gardner,et al.  When Listeners Talk: Response Tokens and Listener Stance , 2001 .

[4]  Judit Kormos.  Simulating conversations in oral-proficiency assessment: a conversation analysis of role plays and non-scripted interviews in language exams , 1999 .

[5]  Annie Brown,et al.  Interviewer Variability in Oral Proficiency Interviews , 2005 .

[6]  Claire J. Kramsch.  From Language Proficiency to Interactional Competence , 1986 .

[7]  Anne Lazaraton,et al.  The Structural Organization of a Language Interview: A Conversation Analytic Perspective. , 1992 .

[8]  John H. A. L. de Jong,et al.  Individualizing the assessment of language abilities , 1992 .

[9]  Anne Lazaraton,et al.  Preference Organization in Oral Proficiency Interviews: The Case of Language Ability Assessments , 1997 .

[10]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[11]  Alison Green,et al.  Verbal Protocol Analysis in Language Testing Research: A Handbook , 1998 .

[12]  Jeff Connor-Linton,et al.  TALKING AND TESTING: DISCOURSE APPROACHES TO THE ASSESSMENT OF ORAL PROFICIENCY.Richard Young and Agnes Weiyun He (Eds.). Amsterdam: Benjamins, 1998. Pp. x + 395. NLG 138 cloth. , 2000, Studies in Second Language Acquisition.

[13]  Barry O'Sullivan,et al.  Using observation checklists to validate speaking-test tasks , 2002 .

[14]  Marysia Johnson.  The Art of Non-conversation , 2008 .

[15]  Dave Robertson,et al.  Towards Objectivity in Group Oral Testing , 1976 .

[16]  Alister Cumming,et al.  Decision Making While Rating ESL/EFL Writing Tasks: A Descriptive Framework. , 2002 .

[17]  J. Norton,et al.  The paired format in the Cambridge Speaking Tests , 2005 .

[18]  Carolyn E. Turner,et al.  Developing rating scales for the assessment of second language performance , 1996 .

[19]  Merrill Swain,et al.  Examining dialogue: another approach to content specification and to validating inferences drawn from test scores , 2001 .

[20]  P. Glover,et al.  Readers respond. Oral testing in pairs - secondary school perspective , 2001 .

[21]  Glenn Fulcher,et al.  Testing Second Language Speaking , 2003 .

[22]  Elinor Ochs,et al.  Co-Construction: An Introduction , 1995 .

[23]  Anne Lazaraton,et al.  Gesture and Speech in the Vocabulary Explanations of One ESL Teacher: A Microanalytic Inquiry. , 2004 .

[24]  Michael H. Long.  Native speaker/non-native speaker conversation and the negotiation of comprehensible input1 , 1983 .

[25]  Lyn May.  An examination of rater orientations on a paired candidate discussion task through stimulated verbal recall , 2006 .

[26]  Anne Lazaraton,et al.  A qualitative approach to the validation of oral language tests , 2002 .

[27]  James Dean Brown,et al.  A focus on language test development : expanding the language proficiency construct across a variety of tests , 2001 .

[28]  Tim McNamara,et al.  Language assessment as social practice: challenges for research , 2001 .

[29]  Annie Brown,et al.  Interviewer variation and the co-construction of speaking proficiency , 2003 .

[30]  Nick Saville,et al.  Assessing speaking in the revised FCE , 1999 .

[31]  Samuel Messick,et al.  Validity and washback in language testing , 1996 .

[32]  Noriko Iwashita,et al.  AN EXAMINATION OF RATER ORIENTATIONS AND TEST-TAKER PERFORMANCE ON ENGLISH-FOR-ACADEMIC-PURPOSES SPEAKING TASKS , 2005 .

[33]  L Taylor,et al.  THE PAIRED SPEAKING TEST FORMAT: RECENT STUDIES , 2001 .

[34]  Glenn Fulcher,et al.  Does thick description lead to smart tests? A data-based approach to rating scale construction , 1996 .

[35]  Arthur Hughes,et al.  Current developments in language testing , 1985 .

[36]  Brian North,et al.  Scaling descriptors for language proficiency scales , 1998 .

[37]  Leo Van Lier,et al.  Reeling, Writhing, Drawling, Stretching, and Fainting in Coils: Oral Proficiency Interviews as Conversation , 1989 .

[38]  Michael Milanovic,et al.  Discourse Variation in Oral Proficiency Interviews , 1992, Studies in Second Language Acquisition.