Is Stance Detection Topic-Independent and Cross-topic Generalizable? - A Reproduction Study

Cross-topic stance detection is the task to automatically detect stances (pro, against, or neutral) on unseen topics. We successfully reproduce state-of-the-art cross-topic stance detection work (Reimers et. al, 2019), and systematically analyze its reproducibility. Our attention then turns to the cross-topic aspect of this work, and the specificity of topics in terms of vocabulary and socio-cultural context. We ask: To what extent is stance detection topic-independent and generalizable across topics? We compare the model’s performance on various unseen topics, and find topic (e.g. abortion, cloning), class (e.g. pro, con), and their interaction affecting the model’s performance. We conclude that investigating performance on different topics, and addressing topic-specific vocabulary and context, is a future avenue for cross-topic stance detection. References Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and Clustering of Arguments with Contextualized Word Embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567–578, Florence, Italy. Association for Computational Linguistics.

[1]  Iryna Gurevych,et al.  Stance Detection Benchmark: How Robust is Your Stance Detection? , 2020, KI - Künstliche Intelligenz.

[2]  Antske Fokkens,et al.  No NLP Task Should be an Island: Multi-disciplinarity for Diversity in News Recommender Systems , 2021, HACKASHOP.

[3]  John W. Du Bois.  The stance triangle , 2007 .

[4]  Antske Fokkens,et al.  Offspring from Reproduction Problems: What Replication Failure Teaches Us , 2013, ACL.

[5]  Walid Magdy,et al.  Stance Detection on Social Media: State of the Art and Trends , 2020, Inf. Process. Manag..

[6]  Iryna Gurevych,et al.  A Retrospective Analysis of the Fake News Challenge Stance-Detection Task , 2018, COLING.

[7]  Wenji Mao,et al.  Modeling Transferable Topics for Cross-Target Stance Detection , 2019, SIGIR.

[8]  Matt Thomas,et al.  Get out the vote: Determining support or opposition from Congressional floor-debate transcripts , 2006, EMNLP.

[9]  Swapna Somasundaran,et al.  Recognizing Stances in Online Debates , 2009, ACL.

[10]  Iryna Gurevych,et al.  Classification and Clustering of Arguments with Contextualized Word Embeddings , 2019, ACL.

[11]  Zhiying Xin,et al.  Stancetaking in Discourse: Subjectivity, Evaluation, Interaction , 2008 .

[12]  Mladen Karan,et al.  Data Set for Stance and Sentiment Analysis from User Comments on Croatian News , 2019, BSNLP@ACL.

[13]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[14]  K. Bretonnel Cohen,et al.  Three Dimensions of Reproducibility in Natural Language Processing , 2018, LREC.

[15]  Nava Tintarev,et al.  Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics , 2021, SIGKDD Explor..

[16]  Same Side Stance Classification , 2020 .

[17]  Kathleen McKeown,et al.  Adversarial Learning for Zero-Shot Stance Detection on Social Media , 2021, NAACL.

[18]  Ehud Reiter,et al.  A Systematic Review of Reproducibility Research in Natural Language Processing , 2021, EACL.

[19]  Preslav Nakov,et al.  Unsupervised User Stance Detection on Twitter , 2019, ICWSM.

[20]  Andreas Vlachos,et al.  Emergent: a novel data-set for stance classification , 2016, NAACL.

[21]  David Schlangen,et al.  Targeting the Benchmark: On Methodology in Current Natural Language Processing Research , 2020, ACL.

[22]  Sameer Singh,et al.  COVIDLies: Detecting COVID-19 Misinformation on Social Media , 2020, NLP4COVID@EMNLP.

[23]  Isabelle Augenstein,et al.  A simple but tough-to-beat baseline for the Fake News Challenge stance detection task , 2017, ArXiv.

[24]  Lixing Zhu,et al.  Hierarchical viewpoint discovery from tweets using Bayesian modelling , 2019, Expert Syst. Appl..

[25]  Iryna Gurevych,et al.  Cross-topic Argument Mining from Heterogeneous Sources , 2018, EMNLP.

[26]  Isabelle Augenstein,et al.  A Survey on Stance Detection for Mis- and Disinformation Identification , 2021, NAACL-HLT.

[27]  Eyal Shnarch,et al.  Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network for Argumentation Mining , 2018, ACL.

[28]  Beata Beigman Klebanov,et al.  Vocabulary Choice as an Indicator of Perspective , 2010, ACL.

[29]  Saif Mohammad,et al.  SemEval-2016 Task 6: Detecting Stance in Tweets , 2016, *SEMEVAL.

[30]  Ralph Bergmann,et al.  Same Side Stance Classification Task: Facilitating Argument Stance Classification by Fine-tuning a BERT Model , 2020, ArXiv.