The Influence of Second Language Experience and Accent Familiarity on Oral Proficiency Rating: A Qualitative Investigation

This study investigates whether raters' knowledge of test takers' first language (L1) affects how the raters orient themselves to the task of rating oral speech. The authors qualitatively investigated the effects of accent familiarity on raters' score assignment processes. Twenty-six trained raters with a second language of Mandarin Chinese, Korean, or Spanish rated English speech samples from 72 English language learners from three L1 backgrounds (Mandarin Chinese, Korean, and Spanish). Afterward, raters watched videos of themselves rating and discussed their rating processes. Analyses of data, assisted by the qualitative analysis software package QSR NVivo 8, revealed that some raters attended to and were influenced by test-taker accent and L1. Raters who were heritage language learners discussed how their personal heritage status influenced how they thought about the speech samples from speakers whose L1 matched their heritage language. Although sensitivity to test-taker accents seemed to occur naturally in the rating process, findings suggest that when raters have learned or know, to varying degrees, the test takers' L1, they tend to orient themselves to the speech in a biased way, compromising test reliability.

[1]  J. Flege.  The detection of French accent by American listeners. , 1984, The Journal of the Acoustical Society of America.

[2]  A. Davies.  Does Language Testing Need the Native Speaker? , 2011 .

[3]  Xiaoming Xi,et al.  Using Raters from India to Score a Large-Scale Speaking Test. , 2011 .

[4]  Dean Brodkey,et al.  DICTATION AS A MEASURE OF MUTUAL INTELLIGIBILITY: A PILOT STUDY , 1972 .

[5]  Tracey M. Derwing,et al.  Second Language Accent and Pronunciation Teaching: A Research- Based Approach. , 2005 .

[6]  John A. Bisazza,et al.  THE COMPREHENSIBILITY OF THREE VARIETIES OF ENGLISH FOR COLLEGE STUDENTS IN SEVEN COUNTRIES1 , 1982 .

[7]  Joan M. Fayer,et al.  Native and Nonnative Judgments of Intelligibility and Irritation , 1987 .

[8]  Tracey M. Derwing,et al.  Foreign Accent, Comprehensibility, and Intelligibility in the Speech of Second Language Learners , 1995 .

[9]  Catherine Elder,et al.  Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs? , 2011 .

[10]  Tessa Bent,et al.  Perceptual adaptation to non-native speech , 2008, Cognition.

[11]  Sara Cushing Weigle,et al.  Effects of training on raters of ESL compositions , 1994 .

[12]  P. Trofimovich,et al.  Intelligibility, Comprehensibility, and Accentedness of L2 Speech: The Role of Listener Experience and Semantic Context , 2008 .

[13]  Kanae Nishi,et al.  Effects of noise and proficiency level on intelligibility of Chinese‐accented English , 2001 .

[14]  Hidetoshi Saito,et al.  EFL classroom peer assessment: Training effects on rating and commenting , 2008 .

[15]  Ann Wennerstrom,et al.  Language Learners' Perceptions of Accent , 2006 .

[16]  Tracey M. Derwing,et al.  THE MUTUAL INTELLIGIBILITY OF L2 SPEECH , 2006, Studies in Second Language Acquisition.

[17]  Jeremy Goslin,et al.  Regional and Foreign Accent Processing in English: Can Listeners Adapt? , 2009, Journal of psycholinguistic research.

[18]  Xiaoming Xi,et al.  HOW DO RATERS FROM INDIA PERFORM IN SCORING THE TOEFL IBT™ SPEAKING SECTION AND WHAT KIND OF TRAINING HELPS? , 2009 .

[19]  Brian K. Lynch,et al.  Investigating variability in tasks and rater judgements in a performance test of foreign language speaking , 1995 .

[20]  Tom Lumley,et al.  Rater characteristics and rater bias: implications for training , 1995 .

[21]  David Birdsong,et al.  Degree of foreign accent in English sentences produced by Korean children and adults , 2006, J. Phonetics.

[22]  Ferenc Bunta,et al.  The Effects of Nonnative Accents on Listening Comprehension: Implications for ESL Assessment , 2002 .

[23]  Sara Cushing Weigle,et al.  Using FACETS to model rater training effects , 1998 .

[24]  M. Bresnahan,et al.  Attitudinal and affective response toward accented English , 2002 .

[25]  Gillian Wigglesworth.  Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction , 1993 .

[26]  Tessa Bent,et al.  The interlanguage speech intelligibility benefit. , 2003, The Journal of the Acoustical Society of America.

[27]  Shawn A. Weil,et al.  Foreign accented speech: Encoding and generalization , 2001 .

[28]  J. Luk,et al.  Accent and Second Language Listening Comprehension , 1997 .

[29]  Tracey M. Derwing,et al.  ACCENT, INTELLIGIBILITY, AND COMPREHENSIBILITY , 1997, Studies in Second Language Acquisition.

[30]  Tessa Bent,et al.  The interlanguage speech intelligibility benefit for native speakers of Mandarin: Production and perception of English word-final voicing contrasts , 2008, J. Phonetics.

[31]  Micheline Chalhoub-Deville,et al.  Deriving oral assessment scales across different tests and rater groups , 1995 .

[32]  R. Major.  Foreign accent : the ontogeny and phylogeny of second language phonology , 2001 .

[33]  Anne Brown,et al.  The effect of rater variables in the development of an occupation-specific language performance test , 1995 .

[34]  L. Nygaard,et al.  Perceptual learning of systematic variation in Spanish-accented speech. , 2009, The Journal of the Acoustical Society of America.

[35]  Evangeline Marlos Varonis,et al.  The Effect Of Familiarity On The Comprehensibility Of Nonnative Speech , 1984 .

[36]  Larry E. Smith,et al.  English for Cross-Cultural Communication: The Question of Intelligibility. , 1979 .

[37]  Youn-hee Kim.  An investigation into native and non-native teachers' judgments of oral English performance: A mixed methods approach , 2009 .

[38]  Robert Mannell,et al.  Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews? , 2011 .

[39]  I. R. MacKay,et al.  Evaluating the effects of chronological age and sentence duration on degree of perceived foreign accent , 2006, Applied Psycholinguistics.

[40]  S. Gass,et al.  The comprehensibility of non-native speech , 1982, Studies in Second Language Acquisition.

[41]  H. Giles,et al.  Language attitudes toward varieties of English: An American‐Japanese context , 1998 .

[42]  David R. Thomas,et al.  A General Inductive Approach for Analyzing Qualitative Evaluation Data , 2006 .

[43]  Tracey M. Derwing,et al.  The functional load principle in ESL pronunciation instruction: An exploratory study , 2006 .

[44]  Paula M. Winke,et al.  The Relationship Between Raters' Prior Language Study and the Evaluation of Foreign Language Speech Samples , 2011 .

[45]  Patterns of rater behaviour in the assessment of an oral interaction test , 1994 .

[46]  Constance M. Clarke,et al.  Rapid adaptation to foreign-accented English. , 2004, The Journal of the Acoustical Society of America.