Assessed Levels of Second Language Speaking Proficiency: How Distinct?

The study reported in this paper is an investigation of the nature of speaking proficiency in English as a second language in the context of a larger project to develop a rating scale for a new international test of English for Academic Purposes, TOEFL iBT (Brown et al. 2005). We report on a large-scale study of the relationship between detailed features of the spoken language produced by test-takers and holistic scores awarded by raters to these performances. Spoken test performances representing five different tasks and five different proficiency levels (200 performances in all) were analyzed using a range of measures of grammatical accuracy and complexity, vocabulary, pronunciation, and fluency. The results showed that features from each category helped distinguish overall levels of performance, with particular features of vocabulary and fluency having the strongest impact. Overall, the study contributes important insights into the nature of spoken proficiency as it develops and can be measured in rating scales for speaking, and has implications for methodological issues of the appropriateness of the use in language testing research contexts of measures developed in research on second language acquisition.

[1]  Lindsay Brooks.  A Tale of Two Tests: A BOOK REVIEW OF The Equivalence of Direct and Semi-direct Speaking Tests. Kieran O'Loughlin. Cambridge University Press, 280 pp., $33.00 , 2006 .

[2]  Yong-Won Lee,et al.  DEPENDABILITY OF NEW ESL WRITING TEST SCORES: EVALUATING PROTOTYPE TASKS AND ALTERNATIVE RATING SCHEMES , 2005 .

[3]  Noriko Iwashita,et al.  AN EXAMINATION OF RATER ORIENTATIONS AND TEST-TAKER PERFORMANCE ON ENGLISH-FOR-ACADEMIC-PURPOSES SPEAKING TASKS , 2005 .

[4]  Tim McNamara,et al.  Discourse and Assessment. , 2002 .

[5]  K. O’Loughlin,et al.  The Equivalence of Direct and Semi-Direct Speaking Tests , 2001 .

[6]  Noriko Iwashita,et al.  Can We Predict Task Difficulty in an Oral Proficiency Test? Exploring the Potential of an Information‐Processing Approach to Task Design , 2001 .

[7]  A. Vermeer.  Coming to grips with lexical richness in spontaneous speech data , 2000 .

[8]  Peter Skehan,et al.  The Influence of Task Structure and Processing Conditions on Narrative Retellings. , 1999 .

[9]  L. Ortega.  PLANNING AND FOCUS ON FORM IN L2 ORAL PERFORMANCE , 1999, Studies in Second Language Acquisition.

[10]  Uta Mehnert,et al.  THE EFFECTS OF DIFFERENT LENGTHS OF TIME FOR PLANNING ON SECOND LANGUAGE PERFORMANCE , 1998, Studies in Second Language Acquisition.

[11]  D. Schallert,et al.  The Relative Contribution of L2 Language Proficiency and L1 Reading Ability to L2 Reading Performance: A Test of the Threshold Hypothesis in an EFL Context , 1997 .

[12]  Gillian Wigglesworth,et al.  An investigation of planning time and proficiency level on oral test discourse , 1997 .

[13]  Peter Skehan,et al.  The Influence of Planning and Task Type on Second Language Performance , 1996, Studies in Second Language Acquisition.

[14]  Glenn Fulcher,et al.  Does thick description lead to smart tests? A data-based approach to rating scale construction , 1996 .

[15]  T. McNamara.  Measuring Second Language Performance , 1996 .

[16]  R. Hawkins,et al.  The development of fluency in advanced learners of French , 1996 .

[17]  Peter Robinson,et al.  Task Complexity and Second Language Narrative Discourse , 1995 .

[18]  N. Denzin,et al.  Handbook of Qualitative Research , 1994 .

[19]  Dan Douglas,et al.  Quantity and quality in speaking test performance , 1994 .

[20]  Elana Shohamy,et al.  The validity of direct versus semi- direct oral tests , 1994 .

[21]  Larry Selinker,et al.  Analyzing oral proficiency test performance in general and specific purpose contexts , 1992 .

[22]  B. MacWhinney.  The CHILDES project: tools for analyzing talk , 1992 .

[23]  T. McNamara.  Item Response Theory and the validation of an ESP test for health professionals , 1990 .

[24]  Leo Van Lier,et al.  Reeling, Writhing, Drawling, Stretching, and Fainting in Coils: Oral Proficiency Interviews as Conversation , 1989 .

[25]  Sally Sieloff Magnan.  Grammar and the ACTFL Oral Proficiency Interview: Discussion and Data , 1988 .

[26]  Heidi Byrnes,et al.  Defining and developing proficiency : guidelines, implementations, and concepts , 1987 .

[27]  Teresa Pica,et al.  Methods of Morpheme Quantification: Their Effect on the Interpretation of Second Language Data , 1983, Studies in Second Language Acquisition.

[28]  M. Swain,et al.  THEORETICAL BASES OF COMMUNICATIVE APPROACHES TO SECOND LANGUAGE TEACHING AND TESTING , 1980 .

[29]  K. W. Hunt.  Syntactic maturity in schoolchildren and adults. , 1970, Monographs of the Society for Research in Child Development.

[30]  Tom Lumley,et al.  Assessing second language writing : the rater's perspective , 2005 .

[31]  Barbara F. Freed.  Is fluency, like beauty, in the eyes (and ears) of the beholder? , 2000 .

[32]  Heidi Riggenbach,et al.  Perspectives on Fluency , 2000 .

[33]  Geoff Brindley,et al.  Interfaces between Second Language Acquisition and Language Testing Research: Describing language development? Rating scales and SLA , 1999 .

[34]  Shunji Inagaki,et al.  Second Language Development in Writing: Measures of Fluency, Accuracy, and Complexity , 1998 .

[35]  A. Strauss,et al.  Grounded theory methodology: An overview. , 1994 .

[36]  Ludo Verhoeven,et al.  The Construct of Language Proficiency. , 1992 .

[37]  Geoff Brindley,et al.  The assessment of second language proficiency : issues and approaches , 1986 .