Validation of a web-based test of ESL pragmalinguistics

Despite increasing interest in interlanguage pragmatics research, research on assessment of this crucial area of second language competence still lags behind assessment of other aspects of learners’ developing second language (L2) competence. This study describes the development and validation of a 36-item web-based test of ESL pragmalinguistics, measuring learners’ offline knowledge of implicatures and routines with multiple-choice questions, and their knowledge of speech acts with discourse completion tests. The test was delivered online to 267 ESL and EFL learners, ranging in proficiency from beginner to advanced. Evidence for construct validity was collected through correlational analyses and comparisons between groups. The effect of browser familiarity was found to be negligible, and learners generally performed as previous research would suggest: their knowledge of speech acts increased with proficiency, as did their knowledge of implicature. Their knowledge of routines, however, was strongly dependent on L2 exposure. Correlations between the sections and factor analysis confirmed that the routines, implicatures, and speech act sections are related but that each has some unique variance. The test was sufficiently reliable and practical, taking an hour to administer and little time to score. Limitations and future research directions are discussed.

[1]  H. Byrnes,et al.  Learning Foreign and Second Languages , 2000 .

[2]  J. House.  Developing Pragmatic Fluency in English as a Foreign Language: Routines and Metapragmatic Awareness , 1996, Studies in Second Language Acquisition.

[3]  G. Kasper,et al.  Pragmatic Development in a Second Language , 2003 .

[4]  A. Cohen,et al.  DEVELOPING A MEASURE OF SOCIOCULTURAL COMPETENCE: THE CASE OF APOLOGY1 , 1981 .

[5]  Patricia Bou Franch.  On Pragmatic Transfer , 1998 .

[6]  Thom Hudson.  Pragmatics in Language Teaching: Indicators for pragmatic instruction: Some quantitative tools , 2001 .

[7]  Kenneth R. Rose,et al.  AN EXPLORATORY CROSS-SECTIONAL STUDY OF INTERLANGUAGE PRAGMATIC DEVELOPMENT , 2000, Studies in Second Language Acquisition.

[8]  James Dean Brown.  Pragmatics in Language Teaching: Pragmatics tests: Different purposes, different tests , 2001 .

[9]  G. Kasper,et al.  Developmental Issues in Interlanguage Pragmatics , 1996, Studies in Second Language Acquisition.

[10]  James Dean Brown,et al.  Developing prototypic measures of cross-cultural pragmatics , 1998 .

[11]  T. Hill,et al.  The development of pragmatic competence in an EFL context , 1997 .

[12]  Jenny A. Thomas.  Meaning in Interaction: An Introduction to Pragmatics , 1995 .

[13]  Carsten Roever,et al.  WEB-BASED LANGUAGE TESTING , 2001 .

[14]  R. Barnard.  Cognition and Second Language Instruction , 2003 .

[15]  Christine Klein-Braley.  A cloze-up on the C-Test: a study in the construct validation of authentic tests , 1985 .

[16]  Kathleen Bardovi-Harlig,et al.  Experimental and Observational Data in the Study of Interlanguage Pragmatics. , 1992 .

[17]  Margaret A. DuFon,et al.  Cross-Linguistic Influence in Indirectness: The Case of English Directives Performed by Native Japanese Speakers. , 1989 .

[18]  Robin C. Scarcella,et al.  On speaking politely in a second language , 1981 .

[19]  Kenneth R. Rose,et al.  Pragmatics in Language Teaching: Name index , 2001 .

[20]  David Crystal,et al.  A dictionary of linguistics and phonetics , 1997 .

[21]  Kenneth R. Rose,et al.  Eliciting Speech Act Data in Japanese: The Effect of Questionnaire Type , 1995 .

[22]  P. Robinson.  Individual differences and instructed language learning , 2002 .

[23]  山下 早代子.  Six measures of JSL pragmatics , 1996 .

[24]  G. Kasper,et al.  Pragmatics in Second Language Learning , 2005 .

[25]  Kristine Billmyer,et al.  Investigating instrument-based pragmatic variability: effects of enhancing discourse completion tests , 2000 .

[26]  R. M. Thorndike.  Measurement and Evaluation in Psychology and Education , 1969 .

[27]  Anna Trosborg,et al.  Apology strategies in natives/non-natives , 1987 .

[28]  Andrea Golato.  Studying Compliment Responses: A Comparison of DCTs and Recordings of Naturally Occurring Talk , 2003 .

[29]  Jenny A. Thomas.  Cross-Cultural Pragmatic Failure , 1983 .

[30]  Lawrence F. Bouton.  Conversational implicature in a second language: learned slowly when not deliberately taught , 1994 .

[31]  Florian Coulmas,et al.  Introduction: Conversational Routine , 1981 .

[32]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .

[33]  Ren Hu-lin.  Pragmatic competence in a second language , 2004 .

[34]  Roland Hausser,et al.  Principles of Pragmatics , 1989 .

[35]  G. Kasper,et al.  Effect of rejoinders in production questionnaires , 1998 .

[36]  C. Röver.  Testing ESL Pragmatics , 2005 .

[37]  James Dean Brown,et al.  Testing in language programs , 1996 .

[38]  Lawrence F. Bouton.  A Cross-cultural Study of Ability to Interpret Implicatures in English. , 1988 .

[39]  R. Schmidt.  Cognition and Second Language Instruction: Attention , 2001 .

[40]  Lyle F. Bachman,et al.  语言测试实践 = Language testing in practice , 1998 .

[41]  Xitao Fan,et al.  Published Studies of Interrater Reliability Often Overestimate Reliability: Computing the Correct Coefficient , 2000 .

[42]  Samuel Messick.  Validity and washback in language testing , 1996 .

[43]  H. H. Clark.  Responding to indirect speech acts , 1979, Cognitive Psychology.

[44]  Christine Klein-Braley.  C-Tests in the context of reduced redundancy testing: an appraisal , 1997 .