Comparing L2 learners’ writing against parallel machine-translated texts: Raters’ assessment, linguistic complexity and errors

Abstract Recent developments in machine translation, such as in Google Translate, may help second language (L2) writers produce texts in the target language according to their intended meaning. The aim of the present study was to examine the role of machine translation (MT) in L2 writing. For this purpose, 66 Korean English as a foreign language (EFL) university learners produced compositions in which writing tasks were counterbalanced in three writing modes (i.e., Direct Writing, Self-Translated Writing, and Machine-Translated Writing). The learners’ writing products were first graded by independent markers and later submitted for computerized text analyses using BNC-COCA 25000, Coh-Metrix, and SynLex to assess linguistic complexity. The texts were also analyzed for types of errors. The results indicate that MT narrowed the difference of writing ability between the skilled and less skilled learners, facilitated learner use of lower frequency words, and produced syntactically more complex sentences. Error analysis showed a reduction in the quantity of grammatical errors when MT was used to aid L2 writing. However, MT-translated compositions contained more mistranslations and a greater number of poor word choices. The results offer pedagogical implications for using MT for L2 writing.

[1]  Ana Niño,et al.  Evaluating the use of machine translation post-editing in the foreign language class , 2008 .

[2]  J. Hulstijn,et al.  Modeling the development of L1 and EFL writing proficiency of secondary school students , 2011 .

[3]  Z. Dörnyei,et al.  Review Article Communication Strategies in a Second Language: Definitions and Taxonomies , 1997 .

[4]  Alta van Rensburg,et al.  Applying Google Translate in a higher education environment: Translation products assessed , 2012 .

[5]  Gert Rijlaarsdam,et al.  L1 use during L2 writing: an empirical study of a complex phenomenon , 2009 .

[6]  Amittai Axelrod,et al.  Application of statistical machine translation to public health information: a feasibility study , 2011, J. Am. Medical Informatics Assoc..

[7]  Michael Kliffer.  An experiment in MT post-editing by a class of intermediate/advanced French majors , 2005, EAMT.

[8]  K. Roehr-Brackin.  Metalinguistic Awareness and Second Language Acquisition , 2018 .

[9]  Cynthia Ducar,et al.  Machine translation and the L2 classroom: Pedagogical solutions for making peace with Google translate , 2018, Foreign Language Annals.

[10]  L. Ortega.  Syntactic Complexity Measures and Their Relationship to L2 Proficiency: A Research Synthesis of College-Level L2 Writing. , 2003 .

[11]  Eva Thue Vold.  Using Machine-Translated Texts to Generate L3 Learners’ Metalinguistic Talk , 2018 .

[12]  A. Hasselgren,et al.  Lexical teddy bears and advanced learners: a study into the ways Norwegian students cope with English vocabulary , 1994 .

[13]  G. Wigglesworth,et al.  Investigating task design in Academic Writing prompts , 2007 .

[14]  Susan M. Gass,et al.  Second Language Research: Methodology and Design , 2021 .

[15]  John Bitchener,et al.  Written corrective feedback in second language acquisition and writing studies , 2011, Language Teaching.

[16]  Hiroe Kobayashi,et al.  Effects of First Language on Second Language Writing: Translation versus Direct Composition† , 1992 .

[17]  Maite Correa.  Leaving the “peer” out of peer-editing: Online translators as a pedagogical tool in the Spanish as a second language classroom. , 2014 .

[18]  The Primary Causes of Article Errors Made by Korean Advanced Learners of English , 2008 .

[19]  Guy Cook,et al.  Translation in Language Teaching: An Argument for Reassessment , 2010 .

[20]  Peter Skehan,et al.  The Influence of Planning and Task Type on Second Language Performance , 1996, Studies in Second Language Acquisition.

[21]  Dana R. Ferris,et al.  Treatment of Error in Second Language Student Writing , 2002 .

[22]  Heejeong Jeong,et al.  Narrative and expository genre effects on students, raters, and performance criteria , 2017 .

[23]  B. Laufer,et al.  Form-focused Instruction in Second Language Vocabulary Learning: A Case for Contrastive Analysis and Translation , 2008 .

[24]  Wenyu Wang,et al.  L1 Use in the L2 Composing Process: An Exploratory Study of 16 Chinese EFL Writers. , 2002 .

[25]  Trena M. Paulus,et al.  The Effect of Peer and Teacher Feedback on Student Writing , 1999 .

[26]  Yuah V. Chon,et al.  EFL Learners’ Knowledge of High-frequency Words in the Comprehension of Idioms: A Boost or a Burden? , 2019 .

[27]  P. Stapleton,et al.  Assessing the accuracy and teachers' impressions of Google Translate: A study of primary L2 writers in Hong Kong , 2019, English for Specific Purposes.

[28]  Michael Carl,et al.  How Does the Post-Editing of Neural Machine Translation Compare with From-Scratch Translation?: A Product and Process Study , 2019 .

[29]  S. Crossley.  Technological disruption in foreign language teaching: The rise of simultaneous machine translation , 2018, Language Teaching.

[30]  K. W. Hunt.  Syntactic maturity in schoolchildren and adults. , 1970, Monographs of the Society for Research in Child Development.

[31]  S. Kol,et al.  Google Translate in Academic Writing Courses? , 2018, The EuroCALL Review.

[32]  Shu-Chiao Tsai,et al.  Using google translate in EFL drafts: a preliminary investigation , 2019, Computer Assisted Language Learning.

[33]  M. Groves,et al.  Friend or foe? google translate in language for academic purposes , 2015 .

[34]  Donald S. Qi,et al.  Exploring the Role of Noticing in a Three-Stage Second Language Writing Task. , 2001 .

[35]  Kazuya Saito.  Advanced Second Language Segmental and Suprasegmental Acquisition , 2018, The Handbook of Advanced Proficiency in Second Language Acquisition.

[36]  Paul Rollinson,et al.  Using peer feedback in the ESL writing class , 2005 .

[37]  J. Norris,et al.  Towards an Organic Approach to Investigating CAF in Instructed SLA: The Case of Complexity , 2009 .

[38]  Philip M. McCarthy,et al.  MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment , 2010, Behavior research methods.

[39]  Sangmin-Michelle Lee,et al.  The impact of using machine translation on EFL students’ writing , 2019, Computer Assisted Language Learning.

[40]  James Hoetker,et al.  The Effects of Systematic Variations in Essay Topics on the Writing Performance of College Freshmen. , 1989 .

[41]  H. Jacobs.  Testing Esl Composition: A Practical Approach , 1981 .

[42]  Dana R. Ferris,et al.  ERROR FEEDBACK IN L2 WRITING CLASSES: HOW EXPLICIT DOES IT NEED TO BE? , 2001 .

[43]  Carey Jewitt,et al.  Technology, literacy and learning: a multimodal approach , 2010 .

[44]  Xiaofei Lu,et al.  Automatic analysis of syntactic complexity in second language writing , 2010 .

[45]  Arthur C. Graesser,et al.  Coh-Metrix , 2011 .

[46]  M. Swain,et al.  Problems in Output and the Cognitive Processes They Generate: A Step Towards Second Language Learning , 1995, Applied Linguistics.

[47]  Maciej Ogrodniczuk,et al.  Measuring Readability of Polish Texts: Baseline Experiments , 2014, LREC.

[48]  Charlene Polio,et al.  Understanding, Evaluating, and Conducting Second Language Writing Research , 2016 .

[49]  Ulf Schütze.  Language Learning and the Brain: Lexical Processing in Second Language Acquisition , 2016 .

[50]  M. Groves,et al.  A double-edged sword: the merits and the policy implications of Google Translate in higher education , 2016 .

[51]  Billy R Woodall,et al.  Language-switching: Using the first language while writing in a second language , 2002 .