BRUMS at SemEval-2020 Task 3: Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity

This paper presents the team BRUMS submission to SemEval-2020 Task 3: Graded Word Similarity in Context. The system utilises state-of-the-art contextualised word embeddings, which have some task-specific adaptations, including stacked embeddings and average embeddings. Overall, the approach achieves good evaluation scores across all the languages, while maintaining simplicity. Following the final rankings, our approach is ranked within the top 5 solutions of each language while preserving the 1st position of Finnish subtask 2.

[1]  Jason Eisner,et al.  Lexical Semantics , 2020, The Handbook of English Linguistics.

[2]  Tao Qin,et al.  Incorporating BERT into Neural Machine Translation , 2020, ICLR.

[3]  Matthew Purver,et al.  CoSimLex: A Resource for Evaluating Graded Word Similarity in Context , 2019, LREC.

[4]  Eiichiro Sumita,et al.  Recycling a Pre-trained BERT Encoder for Neural Machine Translation , 2019, EMNLP.

[5]  Alessandro Lenci,et al.  Cognitive Linguistics , 2019, The Oxford Handbook of Cognitive Sociology.

[6]  Daniel Loureiro,et al.  LIAAD at SemDeep-5 Challenge: Word-in-Context (WiC) , 2019, SemDeep@IJCAI.

[7]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[8]  Roland Vollgraf,et al.  FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP , 2019, NAACL.

[9]  Dipanjan Das,et al.  BERT Rediscovers the Classical NLP Pipeline , 2019, ACL.

[10]  Xuanjing Huang,et al.  How to Fine-Tune BERT for Text Classification? , 2019, CCL.

[11]  Yonatan Belinkov,et al.  Linguistic Knowledge and Transferability of Contextual Representations , 2019, NAACL.

[12]  Yiming Yang,et al.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context , 2019, ACL.

[13]  Roland Vollgraf,et al.  Contextual String Embeddings for Sequence Labeling , 2018, COLING.

[14]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[15]  Nigel Collier,et al.  SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity , 2017, *SEMEVAL.

[16]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[17]  Zhiyuan Liu,et al.  A Unified Model for Word Sense Representation and Disambiguation , 2014, EMNLP.

[18]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[19]  P. Gärdenfors.  The Geometry of Meaning: Semantics Based on Conceptual Spaces , 2014 .

[20]  J. Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[21]  Andrew Y. Ng,et al.  Improving Word Representations via Global Context and Multiple Word Prototypes , 2012, ACL.

[22]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[23]  George A. Miller,et al.  Using a Semantic Concordance for Sense Identification , 1994, HLT.

[24]  Senja Pollak,et al.  SemEval-2020 Task 3: Graded Word Similarity in Context , 2020, SEMEVAL.

[25]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[26]  Luigi Di Caro,et al.  Real Life Application of a Question Answering System Using BERT Language Model , 2019, SIGdial.

[27]  Marcos Zampieri,et al.  BRUMS at HASOC 2019: Deep Learning Models for Multilingual Hate Speech and Offensive Language Identification , 2019, FIRE.

[28]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[29]  Satoshi Sekine,et al.  A survey of named entity recognition and classification , 2007 .