Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training

The goal of stance detection is to determine the viewpoint expressed in a piece of text towards a target. These viewpoints or contexts are often expressed in many different languages depending on the user and the platform, which can be a local news outlet, a social media platform, a news forum, etc. Most research in stance detection, however, has been limited to working with a single language and on a few limited targets, with little work on cross-lingual stance detection. Moreover, non-English sources of labelled data are often scarce and present additional challenges. Recently, large multilingual language models have substantially improved the performance on many non-English tasks, especially such with limited numbers of examples. This highlights the importance of model pre-training and its ability to learn from few examples. In this paper, we present the most comprehensive study of cross-lingual stance detection to date: we experiment with 15 diverse datasets in 12 languages from 6 language families, and with 6 low-resource evaluation settings each. For our experiments, we build on pattern-exploiting training, proposing the addition of a novel label encoder to simplify the verbalisation procedure. We further propose sentiment-based generation of stance data for pre-training, which shows sizeable improvement of more than 6% F1 absolute in low-shot settings compared to several strong baselines.

[1]  Luis Espinosa Anke,et al.  XLM-T: A Multilingual Language Model Toolkit for Twitter , 2021, ArXiv.

[2]  Pavel Král,et al.  Stance and Sentiment in Czech , 2018, Computación y Sistemas.

[3]  Leon Derczynski,et al.  Joint Rumour Stance and Veracity Prediction , 2019, NODALIDA.

[4]  Fabio Petroni,et al.  Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models , 2021, FINDINGS.

[5]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[6]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[7]  Paolo Rosso,et al.  Stance Evolution and Twitter Interactions in an Italian Political Debate , 2018, NLDB.

[8]  Christopher D. Manning,et al.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages , 2020, ACL.

[9]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[10]  Marius Mosbach,et al.  On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines , 2020, ArXiv.

[11]  Emily M. Bender,et al.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 , 2021, FAccT.

[12]  Monojit Choudhury,et al.  The State and Fate of Linguistic Diversity and Inclusion in the NLP World , 2020, ACL.

[13]  Helmut Schmid,et al.  Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification , 2020, COLING.

[14]  Arkaitz Zubiaga,et al.  SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours , 2017, *SEMEVAL.

[15]  Jude Khouja,et al.  Stance Prediction and Claim Verification: An Arabic Perspective , 2020, FEVER.

[16]  Percy Liang,et al.  Prefix-Tuning: Optimizing Continuous Prompts for Generation , 2021, ACL.

[17]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[18]  Hinrich Schutze,et al.  It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners , 2020, NAACL.

[19]  Swapna Somasundaran,et al.  Recognizing Stances in Ideological On-Line Debates , 2010, HLT-NAACL 2010.

[20]  Saif Mohammad,et al.  SemEval-2016 Task 6: Detecting Stance in Tweets , 2016, *SEMEVAL.

[21]  Jianfeng Gao,et al.  Domain Adaptation via Pseudo In-Domain Data Selection , 2011, EMNLP.

[22]  Veselin Stoyanov,et al.  Unsupervised Cross-lingual Representation Learning at Scale , 2019, ACL.

[23]  Danqi Chen,et al.  Making Pre-trained Language Models Better Few-shot Learners , 2021, ACL/IJCNLP.

[24]  Paolo Rosso,et al.  Overview of the Task on Stance and Gender Detection in Tweets on Catalan Independence , 2017, IberEval@SEPLN.

[25]  Muhammad Abdul-Mageed,et al.  AraStance: A Multi-Country and Multi-Domain Dataset of Arabic Stance Detection for Fact Checking , 2021, NLP4IF.

[26]  Guodong Zhou,et al.  Stance detection via sentiment information and neural network model , 2018, Frontiers of Computer Science.

[27]  German Rigau,et al.  Multilingual Stance Detection in Tweets: The Catalonia Independence Corpus , 2020, LREC.

[28]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[29]  Swapna Somasundaran,et al.  Detecting Arguing and Sentiment in Meetings , 2007, SIGdial.

[30]  Hiroaki Hayashi,et al.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing , 2021, ACM Comput. Surv..

[31]  Colin Raffel,et al.  Improving and Simplifying Pattern Exploiting Training , 2021, EMNLP.

[33]  Saif Mohammad,et al.  Detecting Stance in Tweets And Analyzing its Interaction with Sentiment , 2016, *SEMEVAL.

[34]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[35]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[36]  Sebastian Riedel,et al.  Language Models as Knowledge Bases? , 2019, EMNLP.

[37]  ZHIZHONG SU,et al.  A Stance Detection Approach Based on Generalized Autoregressive pretrained Language Model in Chinese Microblogs , 2021, ICMLC.

[38]  Joachim Bingel,et al.  Disembodied Machine Learning: On the Illusion of Objectivity in NLP , 2021, ArXiv.

[39]  Guanghui Qin,et al.  Learning How to Ask: Querying LMs with Mixtures of Soft Prompts , 2021, NAACL.

[40]  Cornelia Caragea,et al.  Multi-Task Stance Detection with Sentiment and Stance Lexicons , 2019, EMNLP.

[41]  Taku Kudo,et al.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing , 2018, EMNLP.

[42]  Vinay Singh,et al.  An English-Hindi Code-Mixed Corpus: Stance Annotation and Baseline System , 2018, ArXiv.

[43]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[44]  Dejing Dou,et al.  A Joint Sentiment-Target-Stance Model for Stance Classification in Tweets , 2016, COLING.

[45]  Leon Derczynski,et al.  Stance Prediction for Russian: Data and Analysis , 2018, SEDA.

[46]  Alexander M. Rush,et al.  How many data points is a prompt worth? , 2021, NAACL.

[47]  Tomas Mikolov,et al.  Bag of Tricks for Efficient Text Classification , 2016, EACL.

[48]  Paolo Rosso,et al.  Multilingual stance detection in social media political debates , 2020, Comput. Speech Lang..

[49]  Yu Zhou,et al.  Overview of NLPCC Shared Task 4: Stance Detection in Chinese Microblogs , 2016, NLPCC/ICCPOL.

[50]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[51]  Paolo Rosso,et al.  SardiStance @ EVALITA2020: Overview of the Task on Stance Detection in Italian Tweets , 2020, EVALITA.

[52]  Barbara Plank,et al.  Learning to select data for transfer learning with Bayesian Optimization , 2017, EMNLP.

[53]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[54]  Brian Lester,et al.  The Power of Scale for Parameter-Efficient Prompt Tuning , 2021, EMNLP.

[55]  Jacob Eisenstein,et al.  Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling , 2019, EMNLP.

[56]  Iryna Gurevych,et al.  Stance Detection Benchmark: How Robust is Your Stance Detection? , 2020, KI - Künstliche Intelligenz.

[57]  Ladislav Lenc,et al.  Detecting Stance in Czech News Commentaries , 2017, ITAT.

[58]  Isabelle Augenstein,et al.  Cross-Domain Label-Adaptive Stance Detection , 2021, EMNLP.

[59]  Preslav Nakov,et al.  Integrating Stance Detection and Fact Checking in a Unified Corpus , 2018, NAACL.

[60]  Iryna Gurevych,et al.  What to Pre-Train on? Efficient Intermediate Task Selection , 2021, EMNLP.

[61]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[62]  Guodong Zhou,et al.  Stance Detection with Hierarchical Attention Network , 2018, COLING.