Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?

This paper reports the findings of an empirical study on ESL/EFL teachers’ evaluation and interpretation of oral English proficiency as elicited by the national College English Test-Spoken English Test (CET-SET) of China. Informed by debates on the issue of native speaker (NS) norms which have become the focus of attention in recent years, this study addresses the question of whether judgments of language proficiency by non-native English speaking (NNES) teachers, who are currently used to assess performance on the CET-SET, correspond to those of native English speaking (NES) teachers or whether the two groups draw on different constructs of oral proficiency. Data for the study were derived from two sources: unguided holistic ratings given by a group of 19 NES and 20 NNES teachers to CET-SET speech samples from 30 test-takers, and written comments to justify the ratings assigned. Results yielded by both quantitative (MFRM) and qualitative analyses of teacher data, revealed no significant difference in raters’ holistic judgments of the speech samples and a broad level of agreement between groups on the construct components of oral English proficiency. However, the analysis of raters’ comments revealed both quantitative and qualitative differences in the way NES and NNES teachers weighed various features of the oral proficiency construct in justifying the decisions made. The paper concludes by considering the implications of the study’s findings for debates about the native speaker norm as the target for language learners and test-takers.

[1]  Sandra Lee McKay.  Teaching English as an International Language , 2002 .

[2]  A. Kirkpatrick,et al.  Chinese pragmatic norms and ‘China English’ , 2002 .

[3]  J. Jenkins.  Which pronunciation norms and models for English as an International Language , 1998 .

[4]  Alison Green,et al.  Verbal Protocol Analysis in Language Testing Research: A Handbook , 1998 .

[5]  Youn-hee Kim.  An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach , 2009 .

[6]  Jennifer Jenkins,et al.  The Phonology of English as an International Language: New Models, New Norms, New Goals , 2000 .

[7]  D. Crystal,et al.  English as a Global Language , 1998 .

[8]  J. Joseph.  Language and Identity: National, Ethnic, Religious , 2004 .

[9]  David Barnwell,et al.  'Naive' native speakers and judgements of oral proficiency in Spanish , 1989 .

[10]  T. Santos.  Professors' Reactions to the Academic Writing of Nonnative-Speaking Students. , 1988 .

[11]  Gary J. Ockey,et al.  A many-facet Rasch analysis of the second language group oral discussion task , 2003 .

[12]  Gillian Wigglesworth,et al.  An investigation of planning time and proficiency level on oral test discourse , 1997 .

[13]  P. Lowenberg.  Issues of validity in tests of English as a world language: whose standards? , 1993 .

[14]  Micheline Chalhoub-Deville,et al.  Deriving oral assessment scales across different tests and rater groups , 1995 .

[15]  Yasukata Yano,et al.  World Englishes in 2000 and Beyond. , 2001 .

[16]  Brian North,et al.  Scaling descriptors for language proficiency scales , 1998 .

[17]  John Robert Ross,et al.  Where's English? , 1979 .

[18]  A. Pennycook.  Global Englishes, Rip Slyme, and performativity , 2003 .

[19]  Tom Lumley,et al.  The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational I settings , 1997 .

[20]  Micheline Chalhoub-Deville,et al.  A contextualized approach to describing oral language proficiency , 1995 .

[21]  Edward W Wolfe,et al.  Detecting and measuring rater effects using many-facet Rasch measurement: part I. , 2003, Journal of applied measurement.

[22]  M. Rampton,et al.  Displacing the ‘native speaker’: expertise, affiliation, and inheritance , 1990 .

[23]  B. Seidlhofer.  Closing A Conceptual Gap: The Case For A Description Of English As A Lingua Franca , 2001 .

[24]  Anne Brown,et al.  The effect of rater variables in the development of an occupation-specific language performance test , 1995 .

[25]  K. Samimy,et al.  Transcending the nativeness paradigm , 2001 .

[26]  Language Testing and English as an International Language Constraints and Contributions , 2008 .

[27]  A. Brown,et al.  An investigation of the rating process in the IELTS oral interview , 2000 .

[28]  B. Wright.  Reasonable mean-square fit values , 1994 .

[29]  Ling Shi,et al.  Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing , 2001 .

[30]  Betsy L. Hadden.  Teacher and Nonteacher Perceptions of Second‐Language Communication* , 1991 .

[31]  Kingsley Bolton,et al.  Chinese Englishes: From Canton Jargon to Global English. , 2002 .

[32]  Alan Davies,et al.  ASSESSING ENGLISH AS A LINGUA FRANCA , 2006, Annual Review of Applied Linguistics.

[33]  Liz Hamp-Lyons,et al.  Research Perspectives on English for Academic Purposes: World Englishes: Issues in and from academic writing assessment , 2001 .

[34]  Noriko Iwashita,et al.  Can We Predict Task Difficulty in an Oral Proficiency Test? Exploring the Potential of an Information‐Processing Approach to Task Design , 2001 .

[35]  M. Berns.  Expanding on the Expanding Circle: where do WE go from here? , 2005 .

[36]  Brian K. Lynch,et al.  Investigating variability in tasks and rater judgements in a performance test of foreign language speaking , 1995 .

[37]  Peter Strevens,et al.  What Is 'Standard English'? , 1981 .

[38]  Brian K. Lynch,et al.  Using G-theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants , 1998 .

[39]  T. McNamara.  Measuring Second Language Performance , 1996 .

[40]  Noriko Iwashita,et al.  AN EXAMINATION OF RATER ORIENTATIONS AND TEST-TAKER PERFORMANCE ON ENGLISH-FOR-ACADEMIC-PURPOSES SPEAKING TASKS , 2005 .

[41]  Alan Davies,et al.  Whose Norms? International Proficiency Tests in English , 2003 .

[42]  V. Cook,et al.  Teachers of English to Speakers of Other Languages , Inc . ( TESOL ) Going beyond the Native Speaker in Language Teaching , 2007 .

[43]  Peter H. Lowenberg,et al.  Assessing English Proficiency in the Expanding Circle. , 2002 .

[44]  T. McNamara,et al.  Rating scales and native speaker performance on a communicatively oriented EAP test , 1993 .

[45]  Sara Cushing Weigle,et al.  Using FACETS to model rater training effects , 1998 .

[46]  Glenn Fulcher,et al.  Testing Second Language Speaking , 2003 .

[47]  Tom McArthur,et al.  Is it world or international or global English, and does it matter? , 2004, English Today.

[48]  Ivor Timmis,et al.  Native‐speaker norms and International English: a classroom view , 2002 .

[49]  Joan M. Fayer,et al.  Native and Nonnative Judgments of Intelligibility and Irritation , 1987 .

[50]  Vicki Galloway.  Perceptions of the Communicative Efforts of American Students of Spanish , 1980 .

[51]  A. Davies.  The Native Speaker in Applied Linguistics , 1997 .

[52]  J. Jenkins.  A sociolinguistically based, empirically researched pronunciation syllabus for English as an international language , 2002 .

[53]  V. Malabonga.  Book Review: The native speaker: myth and reality , 2004 .

[54]  E. Schnitzer.  English as an international language: implications for interculturalists and language educators , 1995 .

[55]  A. Davies.  Standard English: Discordant Voices , 1999 .

[56]  P. Bruthiaux.  Squaring the circles: issues in modeling English worldwide , 2003 .