CL-IMS @ DIACR-Ita: Volente o Nolente: BERT does not outperform SGNS on Semantic Change Detection

We present the results of our participation in the DIACR-Ita shared task on lexical semantic change detection for Italian. We exploit Average Pairwise Distance of token-based BERT embeddings between time points and rank 5 (of 8) in the official ranking with an accuracy of $.72$. While we tune parameters on the English data set of SemEval-2020 Task 1 and reach high performance, this does not translate to the Italian DIACR-Ita data set. Our results show that we do not manage to find robust ways to exploit BERT embeddings in lexical semantic change detection.

[1]  Erik Velldal,et al.  Diachronic word embeddings and semantic shifts: a survey , 2018, COLING.

[2]  Ehsaneddin Asgari,et al.  Unsupervised Embedding-based Detection of Lexical Semantic Changes , 2020, SemEval@COLING.

[3]  Barbara McGillivray,et al.  Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings , 2019, EMNLP.

[4]  Barbara McGillivray,et al.  SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection , 2020, SEMEVAL.

[5]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[6]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[7]  Dominik Schlechtweg,et al.  OP-IMS @ DIACR-Ita: Back to the Roots: SGNS+OP+CD still Rocks Semantic Change Detection , 2020, EVALITA.

[8]  Mario Giulianelli,et al.  Analysing Lexical Semantic Change with Contextualised Word Representations , 2020, ACL.

[9]  Elaine Zosa,et al.  Discovery Team at SemEval-2020 Task 1: Context-sensitive Embeddings Not Always Better than Static for Semantic Change Detection , 2020, SemEval@COLING.

[10]  Lars Borin,et al.  Survey of Computational Approaches to Lexical Semantic Change , 2018, 1811.06278.

[11]  Christin Beck.  DiaSense at SemEval-2020 Task 1: Modeling Sense Change via Pre-trained BERT Embeddings , 2020, SemEval@COLING.

[12]  Eyal Sagi,et al.  Semantic Density Analysis: Comparing Word Meaning across Time and Phonetic Space , 2009 .

[13]  Ondrej Prazák,et al.  UWB @ DIACR-Ita: Lexical Semantic Change Detection with CCA and Orthogonal Transformation , 2020, EVALITA.

[14]  Kawin Ethayarajh,et al.  How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings , 2019, EMNLP.

[15]  Dominik Schlechtweg,et al.  A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains , 2019, ACL.

[16]  Benoît Sagot,et al.  What Does BERT Learn about the Structure of Language? , 2019, ACL.

[17]  Luke S. Zettlemoyer,et al.  Dissecting Contextual Word Embeddings: Architecture and Representation , 2018, EMNLP.

[18]  Dominik Schlechtweg,et al.  Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change , 2018, NAACL.

[19]  Andrey Kutuzov,et al.  UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection , 2020, SEMEVAL.

[20]  Simon Hengchen,et al.  Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change , 2019, ACL.

[21]  Lars Borin,et al.  Survey of Computational Approaches to Diachronic Conceptual Change , 2018, ArXiv.

[22]  Jakub Sido,et al.  UWB at SemEval-2020 Task 1: Lexical Semantic Change Detection , 2020, SEMEVAL.