Capturing Evolution in Word Usage: Just Add More Clusters?

The way the words are used evolves through time, mirroring cultural or technological evolution of society. Semantic change detection is the task of detecting and analysing word evolution in textual data, even in short periods of time. In this paper we focus on a new set of methods relying on contextualised embeddings, a type of semantic modelling that revolutionised the NLP field recently. We leverage the ability of the transformer-based BERT model to generate contextualised embeddings capable of detecting semantic change of words across time. Several approaches are compared in a common setting in order to establish strengths and weaknesses for each of them. We also propose several ideas for improvements, managing to drastically improve the performance of existing approaches.

[1]  Roland Vollgraf,et al.  Contextual String Embeddings for Sequence Labeling , 2018, COLING.

[2]  Jan Snajder,et al.  Leveraging Lexical Substitutes for Unsupervised Word Sense Induction , 2018, AAAI.

[3]  Steven Skiena,et al.  Statistically Significant Detection of Linguistic Change , 2014, WWW.

[4]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[5]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[6]  Marco Baroni,et al.  A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. , 2011, GEMS.

[7]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[8]  Mirella Lapata,et al.  A Bayesian Model of Diachronic Meaning Change , 2016, TACL.

[9]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[10]  Yoav Goldberg,et al.  A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books , 2013, *SEMEVAL.

[11]  Nobuhiro Kaji,et al.  Incremental Skip-gram Model with Negative Sampling , 2017, EMNLP.

[12]  Martin Wattenberg,et al.  Visualizing and Measuring the Geometry of BERT , 2019, NeurIPS.

[13]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[14]  Roger Levy,et al.  Simple dynamic word embeddings for mapping perceptions in the public sphere , 2019, Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science.

[15]  Lars Borin,et al.  Survey of Computational Approaches to Lexical Semantic Change , 2018, 1811.06278.

[16]  Lidia Pivovarova,et al.  Clustering of Russian Adjective-Noun Constructions using Word Embeddings , 2017, BSNLP@EACL.

[17]  Jianxin Li,et al.  Incrementally Learning the Hierarchical Softmax Function for Neural Language Models , 2017, AAAI.

[18]  Patrick Juola,et al.  The Time Course of Language Change , 2003, Comput. Humanit..

[19]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[20]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[21]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[22]  Katrin Erk,et al.  Deep Neural Models of Semantic Shift , 2018, NAACL-HLT.

[23]  Barbara McGillivray,et al.  Room to Glo: A Systematic Comparison of Semantic Change Detection Approaches with Word Embeddings , 2019, EMNLP.

[24]  Lars Borin,et al.  Survey of Computational Approaches to Diachronic Conceptual Change , 2018, ArXiv.

[25]  Yoav Goldberg,et al.  Towards better substitution-based word sense induction , 2019, ArXiv.

[26]  Slav Petrov,et al.  Temporal Analysis of Language through Neural Language Models , 2014, LTCSS@ACL.

[27]  Shen Li,et al.  Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View , 2019, ACL.

[28]  M. Giulianelli.  Lexical Semantic Change Analysis with Contextualised Word Representations , 2019 .

[29]  J. Aitchison.  Language Change: Progress or Decay? , 1981 .

[30]  Xuri Tang,et al.  A state-of-the-art of semantic change computation , 2018, Natural Language Engineering.

[31]  Mario Giulianelli,et al.  Analysing Lexical Semantic Change with Contextualised Word Representations , 2020, ACL.

[32]  Petra Kralj Novak,et al.  Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift , 2020, LREC.

[33]  Stefan Th. Gries,et al.  Assessing frequency changes in multistage diachronic corpora: Applications for historical corpus linguistics and the study of language acquisition , 2009, Lit. Linguistic Comput..

[34]  Sourav S. Bhowmick,et al.  The Past is Not a Foreign Country: Detecting Semantically Similar Terms across Time , 2016, IEEE Transactions on Knowledge and Data Engineering.

[35]  Simon Hengchen,et al.  Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change , 2019, ACL.

[36]  Gregor Wiedemann,et al.  Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings , 2019, KONVENS.

[37]  Erik Velldal,et al.  Diachronic word embeddings and semantic shifts: a survey , 2018, COLING.