Cambridge at SemEval-2021 Task 1: An Ensemble of Feature-Based and Neural Models for Lexical Complexity Prediction

This paper describes our submission to the SemEval-2021 shared task on Lexical Complexity Prediction. We approached it as a regression problem and present an ensemble combining four systems, one feature-based and three neural with fine-tuning, frequency pre-training and multi-task learning, achieving Pearson scores of 0.8264 and 0.7556 on the trial and test sets respectively (sub-task 1). We further present our analysis of the results and discuss our findings.

[1]  Katherine A. Preston.  The Speed of Word Perception and Its Relation to Reading Ability , 1935 .

[2]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[3]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[4]  Helen Yannakoudakis,et al.  Auxiliary Objectives for Neural Error Detection Models , 2017, BEA@EMNLP.

[5]  Lucia Specia,et al.  A Report on the Complex Word Identification Shared Task 2018 , 2018, BEA@NAACL-HLT.

[6]  Matthew Shardlow,et al.  Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline , 2014, LREC.

[7]  Thomas Wolf,et al.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , 2019, ArXiv.

[8]  Benefits of alternative evaluation methods for Automated Essay Scoring , 2021, EDM.

[9]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[10]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[11]  R. P. Fishburne,et al.  Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel , 1975 .

[12]  Marcos Zampieri,et al.  SemEval-2021 Task 1: Lexical Complexity Prediction , 2021, SEMEVAL.

[13]  Helen Yannakoudakis,et al.  Neural and FST-based approaches to grammatical error correction , 2019, BEA@ACL.

[14]  G. Āllport.  The Psycho-Biology of Language. , 1936 .

[15]  M. Coleman,et al.  A computer readability formula designed for machine scoring. , 1975 .

[16]  Michael C. Doyle,et al.  Effects of frequency on visual word recognition tasks: where are they? , 1989, Journal of experimental psychology. General.

[17]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[18]  J. Catlin,et al.  On the word-frequency effect. , 1969 .

[19]  S. Kemper,et al.  Adults' reading comprehension: effects of syntactic complexity and working memory. , 1992, Journal of gerontology.

[20]  Ekaterina Kochmar,et al.  CAMB at CWI Shared Task 2018: Complex Word Identification with Ensemble-Based Voting , 2018, BEA@NAACL-HLT.

[21]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[22]  Lucia Specia,et al.  Complex Word Identification: Challenges in Data Annotation and System Performance , 2017, NLP-TEA@IJCNLP.

[23]  Marcos Zampieri,et al.  CompLex - A New Corpus for Lexical Complexity Predicition from Likert Scale Data , 2020, READI.

[24]  F. Ferreira.  Effects of length and syntactic complexity on initiation times for prepared utterances , 1991 .

[25]  Ekaterina Kochmar,et al.  Recursive Context-Aware Lexical Simplification , 2019, EMNLP.

[26]  Ted Briscoe,et al.  Text Readability Assessment for Second Language Learners , 2016, BEA@NAACL-HLT.

[27]  Ekaterina Kochmar,et al.  Complex Word Identification as a Sequence Labelling Task , 2019, ACL.

[28]  Shiva Taslimipoor,et al.  MTLB-STRUCT @PARSEME 2020: Capturing Unseen Multiword Expressions Using Multi-task Learning and Pre-trained Masked Language Models , 2020, MWE-LEX.

[29]  Lucia Specia,et al.  SemEval 2016 Task 11: Complex Word Identification , 2016, *SEMEVAL.

[30]  Lucia Specia,et al.  SV000gg at SemEval-2016 Task 11: Heavy Gauge Complex Word Identification with System Voting , 2016, SemEval@NAACL-HLT.