Temporally-Informed Analysis of Named Entity Recognition

Natural language processing models often have to make predictions on text data that evolves over time as a result of changes in language use or the information described in the text. However, evaluation results on existing data sets are seldom reported by taking the timestamp of the document into account. We analyze and propose methods that make better use of temporally-diverse training data, with a focus on the task of named entity recognition. To support these experiments, we introduce a novel data set of English tweets annotated with named entities. We empirically demonstrate the effect of temporal drift on performance, and how the temporal information of documents can be used to obtain better models compared to those that disregard temporal information. Our analysis gives insights into why this information is useful, in the hope of informing potential avenues of improvement for named entity recognition as well as other NLP tasks under similar experimental setups.

[1]  Liviu P. Dinu,et al.  Temporal Text Ranking and Automatic Dating of Texts , 2014, EACL.

[2]  Mark Dredze,et al.  Annotating Named Entities in Twitter Data with Crowdsourcing , 2010, Mturk@HLT-NAACL.

[3]  Kalina Bontcheva,et al.  Broad Twitter Corpus: A Diverse Named Entity Recognition Resource , 2016, COLING.

[4]  Jianxin Li,et al.  Time-evolving Text Classification with Deep Neural Networks , 2018, IJCAI.

[5]  金權鎬.  Semantics, An Introduction to The Science of Meaning , 1965 .

[6]  Ming Zhou,et al.  Recognizing Named Entities in Tweets , 2011, ACL.

[7]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[8]  Raphaël Troncy,et al.  Making Sense of Microposts (#Microposts2016) Named Entity rEcognition and Linking (NEEL) Challenge , 2015, #Microposts.

[9]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[10]  Kjetil Nørvåg,et al.  Improving Temporal Language Models for Determining Time of Non-timestamped Documents , 2008, ECDL.

[11]  Oren Etzioni,et al.  Named Entity Recognition in Tweets: An Experimental Study , 2011, EMNLP.

[12]  Jun Zhou,et al.  Cross-Domain Review Helpfulness Prediction Based on Convolutional Neural Networks with Auxiliary Domain Discriminators , 2018, NAACL.

[13]  Kalina Bontcheva,et al.  USFD: Twitter NER with Drift Compensation and Linked Data , 2015, NUT@IJCNLP.

[14]  Satoshi Sekine,et al.  A survey of named entity recognition and classification , 2007 .

[15]  Cristian Danescu-Niculescu-Mizil,et al.  Finding Your Voice: The Linguistic Development of Mental Health Counselors , 2019, ACL.

[16]  Frederick Reiss,et al.  Domain Adaptation of Rule-Based Annotators for Named-Entity Recognition Tasks , 2010, EMNLP.

[17]  Stephen Ullmann,et al.  Semantics: An Introduction to the Science of Meaning , 1962 .

[18]  Roland Vollgraf,et al.  FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP , 2019, NAACL.

[19]  Lemao Liu,et al.  Instance Weighting for Neural Machine Translation Domain Adaptation , 2017, EMNLP.

[20]  Leon Derczynski,et al.  Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition , 2017, NUT@EMNLP.

[21]  Eric P. Xing,et al.  Diffusion of Lexical Change in Social Media , 2012, PloS one.

[22]  Mark Dredze,et al.  Johns Hopkins or johnny-hopkins: Classifying Individuals versus Organizations on Twitter , 2018, PEOPLES@NAACL-HTL.

[23]  Derek Ruths,et al.  Organizations Are Users Too: Characterizing and Detecting the Presence of Organizations on Twitter , 2015, ICWSM.

[24]  Thorsten Brants,et al.  One billion word benchmark for measuring progress in statistical language modeling , 2013, INTERSPEECH.

[25]  Wei Xu,et al.  Multi-task Pairwise Neural Ranking for Hashtag Segmentation , 2019, ACL.

[26]  Nathanael Chambers,et al.  Labeling Documents with Timestamps: Learning from their Time Expressions , 2012, ACL.

[27]  Aron Culotta,et al.  Dependency Tree Kernels for Relation Extraction , 2004, ACL.

[28]  Gerhard Weikum,et al.  diaNED: Time-Aware Named Entity Disambiguation for Diachronic Corpora , 2018, ACL.

[29]  Katrin Weller,et al.  #Microposts2016: 6th Workshop on Making Sense of Microposts: Big things come in small packages , 2016, WWW.

[30]  Jing Wang,et al.  Multi-Domain Named Entity Recognition with Genre-Aware and Agnostic Inference , 2020, ACL.

[31]  Erik Velldal,et al.  Diachronic word embeddings and semantic shifts: a survey , 2018, COLING.

[32]  Roland Vollgraf,et al.  Contextual String Embeddings for Sequence Labeling , 2018, COLING.

[33]  Silviu Cucerzan,et al.  Large-Scale Named Entity Disambiguation Based on Wikipedia Data , 2007, EMNLP.

[34]  Wei Xu,et al.  Bidirectional LSTM-CRF Models for Sequence Tagging , 2015, ArXiv.

[35]  Mary Ting.  Big Things Come in Small Packages , 2012 .

[36]  Jure Leskovec,et al.  Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change , 2016, ACL.

[37]  Trevor Cohn,et al.  A temporal model of text periodicities using Gaussian Processes , 2013, EMNLP.

[38]  Michael J. Paul,et al.  Examining Temporality in Document Classification , 2018, ACL.

[39]  Chong Wang,et al.  Continuous Time Dynamic Topic Models , 2008, UAI.

[40]  Wei Lu,et al.  Neural Adaptation Layers for Cross-domain Named Entity Recognition , 2018, EMNLP.

[41]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[42]  Michael J. Paul,et al.  Neural Temporality Adaptation for Document Classification: Diachronic Word Embeddings and Domain Adaptation Models , 2019, ACL.

[43]  Iryna Gurevych,et al.  Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging , 2017, EMNLP.

[44]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[45]  Steven Skiena,et al.  Statistically Significant Detection of Linguistic Change , 2014, WWW.

[46]  Raphaël Troncy,et al.  Analysis of named entity recognition and linking for tweets , 2014, Inf. Process. Manag..

[47]  Jacob Eisenstein,et al.  Making “fetch” happen: The influence of social and linguistic context on nonstandard word growth and decline , 2018, EMNLP.

[48]  Kalina Bontcheva,et al.  Generalisation in named entity recognition: A quantitative analysis , 2017, Comput. Speech Lang..

[49]  Tim Oates,et al.  We’re Not in Kansas Anymore: Detecting Domain Changes in Streams , 2010, EMNLP.

[50]  Brendan T. O'Connor,et al.  TweetMotif: Exploratory Search and Topic Summarization for Twitter , 2010, ICWSM.

[51]  Jure Leskovec,et al.  No country for old members: user lifecycle and linguistic change in online communities , 2013, WWW.

[52]  Rahul Goel,et al.  The Social Dynamics of Language Change in Online Networks , 2016, SocInfo.

[53]  Guillaume Lample,et al.  Neural Architectures for Named Entity Recognition , 2016, NAACL.

[54]  Ani Nenkova,et al.  Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve , 2020, ArXiv.

[55]  Steven Bethard,et al.  A Survey on Recent Advances in Named Entity Recognition from Deep Learning models , 2018, COLING.

[56]  Aba-Sah Dadzie,et al.  Making Sense of Microposts (#Microposts2014) Named Entity Extraction & Linking Challenge , 2014, #MSM.

[57]  Tom M. Mitchell,et al.  Learning a Compositional Semantics for Freebase with an Open Predicate Vocabulary , 2015, TACL.

[58]  D. Wijaya,et al.  Understanding semantic change of words over centuries , 2011, DETECT '11.

[59]  John D. Lafferty,et al.  Dynamic topic models , 2006, ICML.

[60]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.