Machine learning for learner English

Abstract This paper discusses machine learning techniques for the prediction of Common European Framework of Reference (CEFR) levels in a learner corpus. We summarise the CAp 2018 Machine Learning (ML) competition, a classification task of the six CEFR levels, which map linguistic competence in a foreign language onto six reference levels. The goal of this competition was to produce a machine learning system to predict learners’ competence levels from written productions comprising between 20 and 300 words and a set of characteristics computed for each text extracted from the French component of the EFCAMDAT data (Geertzen et al., 2013). Together with the description of the competition, we provide an analysis of the results and methods proposed by the participants and discuss the benefits of this kind of competition for the learner corpus research (LCR) community. The main findings address the methods used and lexical bias introduced by the task.

[1]  Jill Burstein,et al.  AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0 , 2004 .

[2]  L. Hirschman,et al.  Principles of Evaluation in Natural Language Processing , 2007 .

[3]  Nitin Madnani,et al.  Second Language Acquisition Modeling , 2018, BEA@NAACL-HLT.

[4]  David M. Magerman.  Statistical Decision-Tree Models for Parsing , 1995, ACL.

[5]  Joel R. Tetreault,et al.  A Report on the First Native Language Identification Shared Task , 2013, BEA@NAACL-HLT.

[6]  Virginie Zampa,et al.  Integrating learner corpora and natural language processing: A crucial step towards reconciling technological sophistication and pedagogical effectiveness1 , 2007, ReCALL.

[7]  Jonathan Baxter.  Theoretical Models of Learning to Learn , 2020 .

[8]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[9]  Magali Paquot,et al.  The Cambridge Handbook of Learner Corpus Research: Learner corpora and native language identification , 2015 .

[10]  Malvina Nissim,et al.  Sharing Is Caring: The Future of Shared Tasks , 2017, Computational Linguistics.

[11]  Luna Filipović,et al.  Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework , 2012 .

[12]  Hwee Tou Ng,et al.  The CoNLL-2013 Shared Task on Grammatical Error Correction , 2013, CoNLL Shared Task.

[13]  Adam Kilgarriff,et al.  Helping Our Own: The HOO 2011 Pilot Shared Task , 2011, ENLG.

[14]  Walt Detmar Meurers,et al.  CTAP: A Web-Based Tool Supporting Automatic Complexity Analysis , 2016, CL4LC@COLING 2016.

[15]  A. O'Keeffe,et al.  The English Grammar Profile of learner competence: methodology and key findings , 2017 .

[16]  Magali Paquot,et al.  Quantitative research methods and study quality in learner corpus research , 2015 .

[17]  Akira Murakami,et al.  Modeling Systematicity and Individuality in Nonlinear Second Language Development: The Case of English Grammatical Morphemes , 2016 .

[18]  Paul Thompson,et al.  Learner corpora: looking towards the future , 2013 .

[19]  Nicolas Ballier,et al.  Automatic Treatment and Analysis of Learner Corpus Data , 2013 .

[20]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[21]  Akira Murakami,et al.  Individual variation and the role of L1 in the L2 development of English grammatical morphemes: insights from learner corpora , 2014 .

[22]  Sebastian Thrun,et al.  Learning to Learn , 1998, Springer US.

[23]  Xiaofei Lu,et al.  Computational Methods for Corpus Annotation and Analysis , 2014 .

[24]  Claudia Leacock,et al.  Automated Grammatical Error Detection for Language Learners , 2010, Synthesis Lectures on Human Language Technologies.

[25]  Nicolas Ballier,et al.  Investigating learners' progression in French as a Foreign Language: vocabulary growth and lexical diversity , 2018 .

[26]  Helen Yannakoudakis,et al.  A New Dataset and Method for Automatically Grading ESOL Texts , 2011, ACL.

[27]  Katrin Wisniewski.  Empirical Learner Language and the Levels of the "Common European Framework of Reference". , 2017 .

[28]  Klaus Zechner,et al.  Automated Essay Scoring: Writing Assessment and Instruction , 2010 .

[29]  E. B. Page,et al.  The use of the computer in analyzing student essays , 1968 .

[30]  S. Weigle.  Validation of automated scores of TOEFL iBT tasks against non-test indicators of writing ability: , 2010 .

[31]  Chaitanya Ramineni,et al.  Learner corpora and automated scoring , 2015 .

[32]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Scott Jarvis,et al.  Data mining with learner corpora , 2011 .

[34]  Paula Lissón.  Investigating the use of readability metrics to detect differences in written productions of learners : a corpus-based study , 2017 .

[35]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[36]  Martin J. Russell,et al.  Overview of the 2018 Spoken CALL Shared Task , 2018, INTERSPEECH.

[37]  Walt Detmar Meurers,et al.  The Cambridge Handbook of Learner Corpus Research: Learner corpora and natural language processing , 2015 .

[38]  Gary Lupyan,et al.  Predictors of L2 word learning accuracy: A big data investigation , 2018, CogSci.

[39]  Lei Zhang,et al.  Sentiment Analysis and Opinion Mining , 2017, Encyclopedia of Machine Learning and Data Mining.

[40]  Fiona Barker,et al.  The Cambridge Handbook of Learner Corpus Research: Learner corpora and language testing , 2015 .

[41]  Robert Dale,et al.  HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task , 2012, BEA@NAACL-HLT.

[42]  Sowmya Vajjala,et al.  Automatic CEFR Level Prediction for Estonian Learner Text , 2014 .

[43]  Walt Detmar Meurers,et al.  The MERLIN corpus: Learner language and the CEFR , 2014, LREC.

[44]  Danielle S. McNamara,et al.  Predicting lexical proficiency in language learner texts using computational indices , 2011 .

[45]  Todd M. Gureckis,et al.  Modeling Second-Language Learning from a Psychological Perspective , 2018, BEA@NAACL-HLT.

[46]  Peter A. Flach,et al.  Machine Learning - The Art and Science of Algorithms that Make Sense of Data , 2012 .

[47]  Martin J. Russell,et al.  Overview of the 2017 Spoken CALL Shared Task , 2017, SLaTE.

[48]  David Alfter,et al.  Classification of Swedish learner essays by CEFR levels , 2016 .

[49]  Hwee Tou Ng,et al.  Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English , 2013, BEA@NAACL-HLT.

[50]  Scott Jarvis,et al.  Data mining with learner corpora: Choosing classifiers for L1 detection , 2011 .

[51]  Magali Paquot,et al.  Learner Corpus Research: An interdisciplinary field on the move , 2015 .

[52]  Paula Buttery,et al.  Criterial Features in Learner Corpora: Theory and Illustrations , 2010 .

[53]  Walt Detmar Meurers,et al.  Task Effects on Linguistic Complexity and Accuracy: A Large-Scale Learner Corpus Analysis Employing Natural Language Processing Techniques , 2017 .