Knowledge Enhanced Masked Language Model for Stance Detection

Detecting stance on Twitter is especially challenging because of the short length of each tweet, the continuous coinage of new terminology and hashtags, and the deviation of sentence structure from standard prose. Fine-tuned language models using large-scale in-domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that enhances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguishability and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election.

[1]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[2]  Walid Magdy,et al.  Your Stance is Exposed! Analysing Possible Factors for Stance Detection on Social Media , 2019, Proc. ACM Hum. Comput. Interact..

[3]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[4]  Shalmoli Ghosh,et al.  Stance Detection in Web and Social Media: A Comparative Study , 2019, CLEF.

[5]  Minlie Huang,et al.  SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge , 2020, EMNLP.

[6]  Xinyan Xiao,et al.  SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis , 2020, ACL.

[7]  Xiao Zhang,et al.  pkudblab at SemEval-2016 Task 6 : A Specific Convolutional Neural Network System for Effective Stance Detection , 2016, *SEMEVAL.

[8]  Kevin Crowston,et al.  Amazon Mechanical Turk: A Research Tool for Organizations and Information Systems Scholars , 2012, Shaping the Future of ICT Research.

[9]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[10]  Yoav Shoham,et al.  SenseBERT: Driving Some Sense into BERT , 2019, ACL.

[11]  George Kurian,et al.  Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.

[12]  Guido Zarrella,et al.  MITRE at SemEval-2016 Task 6: Transfer Learning for Stance Detection , 2016, *SEMEVAL.

[13]  Maosong Sun,et al.  ERNIE: Enhanced Language Representation with Informative Entities , 2019, ACL.

[14]  Gerlof Bouma,et al.  Normalized (pointwise) mutual information in collocation extraction , 2009 .

[15]  Masaki Aono,et al.  Tweet Stance Detection Using an Attention based Neural Ensemble Model , 2019, NAACL.

[16]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[17]  Saif Mohammad,et al.  SemEval-2016 Task 6: Detecting Stance in Tweets , 2016, *SEMEVAL.

[18]  Shourya Roy,et al.  Stance classification of multi-perspective consumer health information , 2018, COMAD/CODS.

[19]  Mong-Li Lee,et al.  Author-aware Aspect Topic Sentiment Model to Retrieve Supporting Opinions from Reviews , 2017, EMNLP.

[20]  Saroj Kaushik,et al.  Twitter Stance Detection — A Subjectivity and Sentiment Polarity Inspired Two-Phase Approach , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[21]  Saif Mohammad,et al.  Detecting Stance in Tweets And Analyzing its Interaction with Sentiment , 2016, *SEMEVAL.

[22]  Yu Sun,et al.  ERNIE: Enhanced Representation through Knowledge Integration , 2019, ArXiv.

[23]  Benjamin Van Durme,et al.  Open Domain Targeted Sentiment , 2013, EMNLP.

[24]  Xuanjing Huang,et al.  GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge , 2019, EMNLP.

[25]  Jie Zhou,et al.  SentiX: A Sentiment-Aware Pre-Trained Model for Cross-Domain Sentiment Analysis , 2020, COLING.

[26]  Ceren Budak,et al.  What happened? The Spread of Fake News Publisher Content During the 2016 U.S. Presidential Election , 2019, WWW.

[27]  Walid Magdy,et al.  Stance Detection on Social Media: State of the Art and Trends , 2020, Inf. Process. Manag..

[28]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[29]  Omer Levy,et al.  SpanBERT: Improving Pre-training by Representing and Predicting Spans , 2019, TACL.

[30]  Tiejun Zhao,et al.  Target-dependent Twitter Sentiment Classification , 2011, ACL.

[31]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[32]  Burt L. Monroe,et al.  Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict , 2008, Political Analysis.

[33]  Saif Mohammad,et al.  A Dataset for Detecting Stance in Tweets , 2016, LREC.

[34]  Ruifeng Xu,et al.  Stance Classification with Target-specific Neural Attention , 2017, IJCAI.

[35]  Lidong Bing,et al.  Recurrent Attention Network on Memory for Aspect Sentiment Analysis , 2017, EMNLP.

[36]  Benjamin Roth,et al.  Joint Aspect and Polarity Classification for Aspect-based Sentiment Analysis with End-to-End Neural Networks , 2018, EMNLP.

[37]  Marco Guerini,et al.  Sentiment Analysis: How to Derive Prior Polarities from SentiWordNet , 2013, EMNLP.

[38]  Yang Yang,et al.  A Survey on Opinion Mining: From Stance to Product Aspect , 2019, IEEE Access.

[39]  Diego Marcheggiani,et al.  You Shall Know a User by the Company It Keeps: Dynamic Representations for Social Media Users in NLP , 2019, EMNLP.

[40]  Noah A. Smith,et al.  Narrative framing of consumer sentiment in online restaurant reviews , 2014, First Monday.