A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors

The study reported here examined the evaluation criteria used to assess the proficiency and effectiveness of the language produced in an oral performance test of English conducted in an American university context. Empirical methods were used to analyze qualitatively and quantitatively transcriptions of the Oral English Tests (OET) of 44 prospective Graduate Student Instructors (GSI). The language required to complete the tasks on the test was conceptualized from the functional perspective of transactional and interactional language use as defined by Brown and Yule (1989). Listening comprehension and pronunciation were also analyzed and scored. Stepwise logistic regression was used to determine the extent to which these linguistic features contributed to final ratings. These quantitative findings were then compared to ‘real-time’ written comments made by evaluators during the tests. Intuitive methods were then used to further explore those features of candidate performance attended to by evaluators: interviews were conducted with experienced evaluators to determine the features they judged necessary for communicating effectively in instructional settings. Results indicate that the three data sources converge on two main features — pronunciation and listening comprehension — that are important in describing and evaluating the proficiency of prospective GSIs.

[1]  G. Kasper,et al.  Cross-Cultural Pragmatics: Requests and Apologies , 1991 .

[2]  Carol A. Chapelle,et al.  VALIDITY IN LANGUAGE ASSESSMENT , 1999, Annual Review of Applied Linguistics.

[3]  Barbara S. Plakans.  Undergraduates' Experiences with and Attitudes Toward International Teaching Assistants. , 1997 .

[4]  Naoko Taguchi.  Task Difficulty in Oral Speech Act Production , 2007 .

[5]  Glenn Fulcher,et al.  Task difficulty in speaking tests , 2003 .

[6]  Carolyn E. Turner,et al.  Constructing rating scales for second language tests , 1995 .

[7]  Fumiyo Nakatsuhara.  Inter-interviewer variation in oral interview tests , 2007 .

[8]  Merrill Swain,et al.  Examining dialogue: another approach to content specification and to validating inferences drawn from test scores , 2001 .

[9]  Carolyn G. Madden,et al.  Discourse and Performance of International Teaching Assistants , 1994 .

[10]  Merrill Swain,et al.  Researching Pedagogic Tasks: Second Language Learning, Teaching, and Testing , 2013 .

[11]  T. McNamara,et al.  Assessed Levels of Second Language Speaking Proficiency: How Distinct? , 2007 .

[12]  Glenn Fulcher,et al.  Does thick description lead to smart tests? A data-based approach to rating scale construction , 1996 .

[13]  Michael Halliday,et al.  Cohesion in English , 1976 .

[14]  Willem J. M. Levelt,et al.  The speaker’s linearization problem , 1981 .

[15]  Dan Douglas,et al.  Assessing Languages for Specific Purposes , 1999 .

[16]  Carolyn E. Turner,et al.  Systematic effects in the rating of second-language speaking ability: test method and learner discourse , 1999 .

[17]  Alan Tonkyn,et al.  Measuring spoken language: a unit for all reasons , 2000 .

[18]  Micheline Chalhoub-Deville,et al.  Deriving oral assessment scales across different tests and rater groups , 1995 .

[19]  T. Lumley,et al.  performance in tape-mediated assessment of speaking The effect of test-taker gender , audience and topic on task , 2006 .

[20]  Martin Bygate,et al.  Quality of language and purpose of task: patterns of learners’ language on two oral communication tasks , 1999 .

[21]  S. Reeves,et al.  Discourse Analysis , 2018, The Study of Language.

[22]  Annie Brown,et al.  Interviewer variation and the co-construction of speaking proficiency , 2003 .

[23]  Li-Shih Huang,et al.  Fine-Tuning the Craft of Teaching by Discussion , 2005 .

[24]  C. Chapelle.  Construct definition and validity inquiry in SLA research , 1999 .

[25]  Micheline Chalhoub-Deville,et al.  Second language interaction: current perspectives and future trends , 2003 .

[26]  Glenn Fulcher,et al.  Testing Second Language Speaking , 2003 .

[27]  K. O’Loughlin,et al.  The impact of gender in oral proficiency testing , 2002 .