IDENTIFYING SPEECH ACTS IN E-MAILS: TOWARD AUTOMATED SCORING OF THE TOEIC® E-MAIL TASK

This study proposes an approach to automatically score the TOEIC® Writing e-mail task. We focus on one component of the scoring rubric, which notes whether the test-takers have used particular speech acts such as requests, orders, or commitments. We developed a computational model for automated speech act identification and tested it on a corpus of TOEIC responses, achieving up to 79.28% accuracy. This model represents a positive first step toward the development of a more comprehensive scoring model. We also created a corpus of speech act-annotated native English workplace e-mails. Comparisons between these and the TOEIC data allow us to assess whether English learners are approximating native models and whether differences between native and non-native data can have negative consequences in the global workplace.

[1]  William B. Stiles,et al.  Describing talk : a taxonomy of verbal response modes , 1992 .

[2]  Stephen G. Pulman,et al.  Information Extraction and Machine Learning: Auto-Marking Short Free Text Responses to Science Questions , 2005, AIED.

[3]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[4]  Mitchell P. Marcus,et al.  Maximum entropy models for natural language ambiguity resolution , 1998 .

[5]  Gabriele Kasper,et al.  PRAGMATICS AND SLA , 1999, Annual Review of Applied Linguistics.

[6]  Jack C. Richards,et al.  SPEECH ACTS AND SECOND LANGUAGE LEARNING , 1980 .

[7]  Donald E. Powers,et al.  AUTOMATED SCORING OF SHORT‐ANSWER OPEN‐ENDED GRE® SUBJECT TEST ITEMS , 2008 .

[8]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[9]  Z. Dörnyei,et al.  Do Language Learners Recognize Pragmatic Violations? Pragmatic Versus Grammatical Awareness in Instructed L2 Learning. , 1998 .

[10]  Steven Bird,et al.  NLTK: The Natural Language Toolkit , 2002, ACL 2006.

[11]  James R. Curran,et al.  Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models , 2007, Computational Linguistics.

[12]  Dale A. Koike.  Pragmatic Competence and Adult L2 Acquisition: Speech Acts in Interlanguage , 1989 .

[13]  Jack C. Richards,et al.  Longman Dictionary of Language Teaching and Applied Linguistics , 1992 .

[14]  Kasia M. Jaszczolt.  Semantics and Pragmatics: Meaning in Language and Discourse , 2002 .

[15]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[16]  Tom M. Mitchell,et al.  Learning to Classify Email into “Speech Acts” , 2004, EMNLP.

[17]  Nancy Ide.  American National Corpus (ANC) , 2002 .

[18]  Zoltán Dörnyei,et al.  On the Teachability of Communication Strategies , 1995 .

[19]  Johanna D. Moore,et al.  Automatic annotation of context and speech acts for dialogue corpora , 2009, Natural Language Engineering.

[20]  Ron Artstein,et al.  Survey Article: Inter-Coder Agreement for Computational Linguistics , 2008, CL.

[21]  Sigrun Biesenbach-Lucas,et al.  Communication topics and strategies in e-mail consultation: Comparison between american and international university students , 2005 .

[22]  J. Ardila.  Meaning in Language. An Introduction to Semantics and Pragmatics , 2011 .

[23]  G. Kasper.  Four perspectives on L2 pragmatic development , 2001 .

[24]  John Blitzer,et al.  Intelligent email: reply and attachment prediction , 2008, IUI '08.

[25]  G. Kasper.  Pragmatic Comprehension in Learner-Native Speaker Discourse. , 1984 .

[26]  M. Swain,et al.  THEORETICAL BASES OF COMMUNICATIVE APPROACHES TO SECOND LANGUAGE TEACHING AND TESTING , 1980 .

[27]  Jan Noyes,et al.  Toward a stochastic speech act model of email behavior , 2008, CEAS.

[28]  Jianfeng Gao,et al.  Using Contextual Speller Techniques and Language Modeling for ESL Error Correction , 2008, IJCNLP.

[29]  Jenny A. Thomas.  Cross-Cultural Pragmatic Failure , 1983 .

[30]  J. Sadock.  Speech acts , 2007 .

[31]  Cécile Paris,et al.  The nature of requests and commitments in email messages , 2008, AAAI 2008.

[32]  John A. Carroll,et al.  Applied morphological processing of English , 2001, Natural Language Engineering.

[33]  Johan Bos,et al.  Linguistically Motivated Large-Scale NLP with C&C and Boxer , 2007, ACL.

[34]  Rachele De Felice,et al.  A Classifier-Based Approach to Preposition and Determiner Error Correction in L2 English , 2008, COLING.

[35]  Jade Goldstein-Stewart,et al.  Using Speech Acts to Categorize Email and Identify Email Genres , 2006, Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06).

[36]  James R. Curran,et al.  Language Independent NER using a Maximum Entropy Tagger , 2003, CoNLL.

[37]  Penelope Brown,et al.  Politeness: Some Universals in Language Usage , 1989 .

[38]  Andrew D. Cohen,et al.  Teaching and assessing L2 pragmatics: What can we expect from learners? , 2008, Language Teaching.

[39]  A. Jaffe.  Stance: Sociolinguistic Perspectives , 2009 .

[40]  Kyu-Hwan Moon.  Speech act study: Differences between native and nonnative speaker complaint strategies , 2002 .

[41]  William W. Cohen,et al.  On the collective classification of email "speech acts" , 2005, SIGIR '05.

[42]  Andrew Lampert,et al.  Can Requests-for-Action and Commitments-to-Act be Reliably Identified in Email Messages ? , 2007 .

[43]  Steven Bird,et al.  NLTK: The Natural Language Toolkit , 2002, ACL.

[44]  Jacob L. Mey,et al.  Between culture and pragmatics: Scylla and Charybdis? The precarious condition of intercultural pragmatics , 2004 .

[45]  William W. Cohen,et al.  Improving “Email Speech Acts” Analysis via N-gram Selection , 2006, HLT-NAACL 2006.

[46]  木村 和夫.  Pragmatics , 1997, Language Teaching.

[47]  Yorick Wilks,et al.  Routing email automatically by purpose not topic , 1999, Natural Language Engineering.

[48]  Csr Young,et al.  How to Do Things With Words , 2009 .

[49]  Jana Z. Sukkarieh,et al.  Leveraging C-Rater's Automated Scoring Capability for Providing Instructional Feedback for Short Constructed Responses , 2008, Intelligent Tutoring Systems.

[50]  Michael Gamon,et al.  Task-Focused Summarization of Email , 2004 .

[51]  Douglas Biber,et al.  Variation across speech and writing: Methodology , 1988 .

[52]  John Blitzer,et al.  Intelligent Email: Aiding Users with AI , 2008, AAAI.

[53]  Martin Chodorow,et al.  CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays , 2003, IAAI.

[54]  Cécile Paris,et al.  Requests and Commitments in Email are More Complex Than You Think: Eight Reasons to be Cautious , 2008, ALTA.

[55]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[56]  John Sie Yuen Lee.  Automatic correction of grammatical errors in non-native English text , 2009 .

[57]  Martin Chodorow,et al.  The Ups and Downs of Preposition Error Detection in ESL Writing , 2008, COLING.

[58]  Cécile Paris,et al.  Classifying Speech Acts using Verbal Response Modes , 2006, ALTA.

[59]  J. Searle,et al.  Expression and Meaning. , 1982 .

[60]  Stephen G. Pulman,et al.  Automatic Short Answer Marking , 2005, ACL 2005.

[61]  Claudia Leacock,et al.  Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications , 2008 .

[62]  R. Kaplan.  CULTURAL THOUGHT PATTERNS IN INTER‐CULTURAL EDUCATION , 1966 .