Diachronic word embeddings and semantic shifts: a survey

Recent years have witnessed a surge of publications aimed at tracing temporal changes in lexical semantics using distributional methods, particularly prediction-based word embedding models. However, this vein of research lacks the cohesion, common terminology and shared practices of more established areas of natural language processing. In this paper, we survey the current state of academic research related to diachronic word embeddings and semantic shifts detection. We start with discussing the notion of semantic shifts, and then continue with an overview of the existing methods for tracing such time-related shifts with word embedding models. We propose several axes along which these methods can be compared, and outline the main challenges before this emerging subfield of NLP, as well as prospects and possible applications.

[1]  Christian Rauh,et al.  Reading Between the Lines: Prediction of Political Violence Using Newspaper Text , 2016, American Political Science Review.

[2]  Richard B. Dasher,et al.  Regularity in Semantic Change: Index of languages , 2001 .

[3]  Slav Petrov,et al.  Temporal Analysis of Language through Neural Language Models , 2014, LTCSS@ACL.

[4]  Dirk Geeraerts,et al.  Diachronic Prototype Semantics: A Contribution to Historical Lexicology , 1997 .

[5]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[6]  Hans-Jörg Schmid,et al.  The NeoCrawler: identifying and retrieving neologisms from the internet and monitoring ongoing change , 2011 .

[7]  Steven Skiena,et al.  Statistically Significant Detection of Linguistic Change , 2014, WWW.

[8]  Gustaf Stern,et al.  Meaning and Change of Meaning: with Special Reference to the English Language , 1975 .

[9]  Patrick Juola,et al.  The Time Course of Language Change , 2003, Comput. Humanit..

[10]  Maarten Marx,et al.  UvA-DARE (Digital Academic Repository) Words are Malleable: Computing Semantic Shifts in Political and Media Discourse , 2017 .

[11]  Andreas Niekler,et al.  Modeling the dynamics of domain specific terminology in diachronic corpora , 2017, ArXiv.

[12]  Björn-Olav Dozo,et al.  Quantitative Analysis of Culture Using Millions of Digitized Books , 2010 .

[13]  Georgiana Dinu,et al.  Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors , 2014, ACL.

[14]  Andrey Kutuzov,et al.  Exploration of register-dependent lexical semantics using word embeddings , 2016, LT4DH@COLING.

[15]  Beate Hampe,et al.  Germanic Future Constructions. A Usage-Based Approach to Language Change , 2009 .

[16]  Hui Xiong,et al.  Dynamic Word Embeddings for Evolving Semantic Discovery , 2017, WSDM.

[17]  Rada Mihalcea,et al.  Word Epoch Disambiguation: Finding How Words Change Over Time , 2012, ACL.

[18]  Stephan Mandt,et al.  Dynamic Word Embeddings , 2017, ICML.

[19]  Jure Leskovec,et al.  Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change , 2016, ACL.

[20]  Felix Hill,et al.  SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation , 2014, CL.

[21]  Marco Baroni,et al.  A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. , 2011, GEMS.

[22]  Gerhard Heyer,et al.  Change of Topics over Time - Tracking Topics by their Change of Meaning , 2009, KDIR.

[23]  Kira Radinsky,et al.  Learning Word Relatedness over Time , 2017, EMNLP.

[24]  S. Gries.  Particle Movement: a Cognitive and Functional Approach* , 2022 .

[25]  Guang Cheng,et al.  Analysing the Semantic Change Based on Word Embedding , 2016, NLPCC/ICCPOL.

[26]  J. R. Firth,et al.  A Synopsis of Linguistic Theory, 1930-1955 , 1957 .

[27]  H. Varian,et al.  Predicting the Present with Google Trends , 2009 .

[28]  Andrew McCallum,et al.  Topics over time: a non-Markov continuous-time model of topical trends , 2006, KDD '06.

[29]  Yang Xu,et al.  A Computational Evaluation of Two Laws of Semantic Change , 2015, CogSci.

[30]  Eyal Sagi,et al.  Tracing semantic change with latent semantic analysis , 2011 .

[31]  W. A. Taylor.  Change-Point Analysis : A Powerful New Tool For Detecting Changes , 2000 .

[32]  Omer Levy,et al.  Neural Word Embedding as Implicit Matrix Factorization , 2014, NIPS.

[33]  Erik Velldal,et al.  Tracing armed conflicts with diachronic word embedding models , 2017, NEWS@ACL.

[34]  Alexander Mehler,et al.  On the Linearity of Semantic Change: Investigating Meaning Variation via Dynamic Graph Models , 2016, ACL.

[35]  Jure Leskovec,et al.  Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change , 2016, EMNLP.

[36]  Nobuhiro Kaji,et al.  Incremental Skip-gram Model with Negative Sampling , 2017, EMNLP.

[37]  Sourav S. Bhowmick,et al.  Omnia Mutantur, Nihil Interit: Connecting Past with Present by Finding Corresponding Terms across Time , 2015, ACL.

[38]  Stefan Th. Gries,et al.  Assessing frequency changes in multistage diachronic corpora: Applications for historical corpus linguistics and the study of language acquisition , 2009, Lit. Linguistic Comput..

[39]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[40]  Sourav S. Bhowmick,et al.  The Past is Not a Foreign Country: Detecting Semantically Similar Terms across Time , 2016, IEEE Transactions on Knowledge and Data Engineering.

[41]  D. Wijaya,et al.  Understanding semantic change of words over centuries , 2011, DETECT '11.

[42]  John D. Lafferty,et al.  Dynamic topic models , 2006, ICML.

[43]  俊雄 齊藤,et al.  The Helsinki Corpus of English Texts と初期近代英語研究 , 1992 .

[44]  Terttu Nevalainen,et al.  CEECing the baseline: lexical stability and significant change in a historical corpus , 2012 .

[45]  Emmerich Kelih.  Quantitative Approaches to the Russian Language , 2020, J. Quant. Linguistics.

[46]  Daphna Weinshall,et al.  Verbs change more than nouns: a bottom-up computational approach to semantic change , 2016 .

[47]  Christopher D. Manning.  Computational Linguistics and Deep Learning , 2015, Computational Linguistics.

[48]  Erik Velldal,et al.  Temporal dynamics of semantic relations in word embeddings: an application to predicting armed conflict participants , 2017, EMNLP.

[49]  Jure Leskovec,et al.  Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora , 2016, EMNLP.

[50]  Carlo Strapparava,et al.  SemEval 2015, Task 7: Diachronic Text Evaluation , 2015, *SEMEVAL.

[51]  J. Deese.  Meaning and change of meaning. , 1967, The American psychologist.

[52]  Andreas Blank,et al.  Historical Semantics and Cognition , 1999 .

[53]  Anders Holst,et al.  Random indexing of text samples for latent semantic analysis , 2000 .

[54]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[55]  Mirella Lapata,et al.  A Bayesian Model of Diachronic Meaning Change , 2016, TACL.

[56]  Andrey Kutuzov,et al.  Two centuries in two thousand words , 2017 .

[57]  J. Bullinaria,et al.  Extracting semantic representations from word co-occurrence statistics: A computational study , 2007, Behavior research methods.

[58]  Yulia Tsvetkov,et al.  A bottom up approach to category mapping and meaning change , 2015, NetWordS.

[59]  Keith Stevens,et al.  Event Detection in Blogs using Temporal Random Indexing , 2009 .

[60]  Daphna Weinshall,et al.  Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models , 2017, EMNLP.

[61]  Katrin Erk,et al.  Deep Neural Models of Semantic Shift , 2018, NAACL-HLT.

[62]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[63]  Anton Osokin,et al.  Breaking Sticks and Ambiguities with Adaptive Skip-gram , 2015, AISTATS.

[64]  C. Habel,et al.  Language , 1931, NeuroImage.

[65]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[66]  Terrence Szymanski,et al.  Temporal Word Analogies: Identifying Lexical Replacement with Diachronic Word Embeddings , 2017, ACL.

[67]  Christian Biemann,et al.  That’s sick dude!: Automatic identification of word sense change across different timescales , 2014, ACL.

[68]  Jianxin Li,et al.  Incrementally Learning the Hierarchical Softmax Function for Neural Language Models , 2017, AAAI.