SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles

We present the results and the main findings of SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. The task featured two subtasks. Subtask SI is about Span Identification: given a plain-text document, spot the specific text fragments containing propaganda. Subtask TC is about Technique Classification: given a specific text fragment, in the context of a full document, determine the propaganda technique it uses, choosing from an inventory of 14 possible propaganda techniques. The task attracted a large number of participants: 250 teams signed up to participate and 44 made a submission on the test set. In this paper, we present the task, analyze the results, and discuss the system submissions and the methods they used. For both subtasks, the best systems used pre-trained Transformers and ensembles.

[1]  Eunsol Choi,et al.  Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking , 2017, EMNLP.

[2]  Ashutosh Modi,et al.  newsSweeper at SemEval-2020 Task 11: Context-Aware Rich Feature Representations for Propaganda Classification , 2020, SEMEVAL.

[3]  Alan W Black,et al.  LTIatCMU at SemEval-2020 Task 11: Incorporating Multi-Level Features for Multi-Granular Propaganda Span Identification , 2020, SEMEVAL.

[4]  Preslav Nakov,et al.  A Survey on Computational Propaganda Detection , 2020, IJCAI.

[5]  Preslav Nakov,et al.  Overview of CheckThat! 2020 English: Automatic Identification and Verification of Claims in Social Media , 2020, CLEF.

[6]  Sinan Aral,et al.  The spread of true and false news online , 2018, Science.

[7]  Preslav Nakov,et al.  Overview of CheckThat! 2020i Arabic: Automatic Identification and Verification of Claims in Social Media , 2020, CLEF.

[8]  Steven Bethard,et al.  TTUI at SemEval-2020 Task 11: Propaganda Detection with Transfer Learning and Ensembles , 2020, SemEval@COLING.

[9]  John. Hunter.  Brainwashing in a large group awareness training? : the classical conditioning hypothesis of brainwashing. , 2015 .

[10]  Preslav Nakov,et al.  Proppy: A System to Unmask Propaganda in Online News , 2019, AAAI.

[11]  Preslav Nakov,et al.  Experiments in Detecting Persuasion Techniques in the News , 2019, ArXiv.

[12]  Xin Tao,et al.  YNUtaoxin at SemEval-2020 Task 11: Identification Fragments of Propaganda Technique by Neural Sequence Labeling Models with Different Tagging Schemes and Pre-trained Language Model , 2020, SemEval@COLING.

[13]  Iryna Gurevych,et al.  Argotario: Computational Argumentation Meets Serious Games , 2017, EMNLP.

[14]  Raj Kumar Gupta,et al.  CrystalFeel at SemEval-2018 Task 1: Understanding and Detecting Emotion Intensity using Affective Lexicons , 2018, *SEMEVAL.

[15]  Samuel Pecar,et al.  NLFIIT at SemEval-2020 Task 11: Neural Network Architectures for Detection of Propaganda Techniques in News Articles , 2020, SemEval@COLING.

[16]  Lei Gao,et al.  Recognizing Explicit and Implicit Hate Speech Using a Weakly Supervised Two-path Bootstrapping Approach , 2017, IJCNLP.

[17]  Preslav Nakov,et al.  Proppy: Organizing the news based on their propagandistic content , 2019, Inf. Process. Manag..

[18]  Verena Blaschke,et al.  CyberWallE at SemEval-2020 Task 11: An Analysis of Feature Engineering for Ensemble Models for Propaganda Detection , 2020, SEMEVAL.

[19]  Brent Brossmann.  Fallacies and Argument Appraisal , 2006 .

[20]  Anna Glazkova,et al.  UTMN at SemEval-2020 Task 11: A Kitchen Solution to Automatic Propaganda Detection , 2020, SEMEVAL.

[21]  Preslav Nakov,et al.  Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 2: Factuality , 2018, CLEF.

[22]  Uuf Brajawidagda,et al.  Tweeting propaganda, radicalization and recruitment: Islamic state supporters multi-sided twitter networks , 2015, DG.O.

[23]  Douglas Walton.  Methods of Argumentation: The Straw Man Fallacy , 2013 .

[24]  Ajay Jaiswal,et al.  Solomon at SemEval-2020 Task 11: Ensemble Architecture for Fine-Tuned Propaganda Detection in News Articles , 2020, SemEval@COLING.

[25]  Preslav Nakov,et al.  Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 1: Check-Worthiness , 2019, CLEF.

[26]  Hiroaki Ozaki,et al.  Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection , 2020, SEMEVAL.

[27]  Preslav Nakov,et al.  Aschern at SemEval-2020 Task 11: It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning , 2020, SEMEVAL.

[28]  Anthony Weston,et al.  A Rulebook for Arguments , 1987 .

[29]  Preslav Nakov,et al.  Overview of CheckThat 2020: Automatic Identification and Verification of Claims in Social Media , 2020, CLEF.

[30]  Vladimir Ivanov,et al.  Inno at SemEval-2020 Task 11: Leveraging Pure Transformer for Multi-Class Propaganda Detection , 2020, SemEval@COLING.

[31]  Robert Jackall.  How to Detect Propaganda , 1995 .

[32]  Ivan Smurov,et al.  NoPropaganda at SemEval-2020 Task 11: A Borrowed Approach to Sequence Tagging and Text Classification , 2020, SEMEVAL.

[33]  Sibel Adali,et al.  Sampling the News Producers: A Large News and Feature Data Set for the Study of the Complex Media Landscape , 2018, ICWSM.

[34]  Arkaitz Zubiaga,et al.  SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours , 2017, *SEMEVAL.

[35]  Vlad Ermurachi,et al.  UAIC1860 at SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles , 2020, SEMEVAL.

[36]  Benno Stein,et al.  Before Name-Calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation , 2018, NAACL.

[37]  Preslav Nakov,et al.  Prta: A System to Support the Analysis of Propaganda Techniques in the News , 2020, ACL.

[38]  Filip Grali'nski,et al.  ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them , 2020, SemEval@COLING.

[39]  Renee Hobbs,et al.  Teaching about Propaganda: An Examination of the Historical Roots of Media Literacy , 2014, Journal of Media Literacy Education.

[40]  Andreas Vlachos,et al.  Automated Fact Checking: Task Formulations, Methods and Future Directions , 2018, COLING.

[41]  Preslav Nakov,et al.  Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 2: Evidence and Factuality , 2019, CLEF.

[42]  Gabriel H. Teninbaum.  Reductio ad Hitlerum: Trumping the Judicial Nazi Card , 2009 .

[43]  Qiaozhu Mei,et al.  UMSIForeseer at SemEval-2020 Task 11: Propaganda Detection by Fine-Tuning BERT with Resampling and Ensemble Learning , 2020, SemEval@COLING.

[44]  Aniruddha Chauhan,et al.  PsuedoProp at SemEval-2020 Task 11: Propaganda Span Detection Using BERT-CRF and Ensemble Sentence Level Classifier , 2020, SemEval@COLING.

[45]  Will Styler,et al.  Anafora: A Web-based General Purpose Annotation Tool , 2013, HLT-NAACL.

[46]  Preslav Nakov,et al.  Fine-Grained Analysis of Propaganda in News Article , 2019, EMNLP.

[47]  Dumitru-Clementin Cercel,et al.  UPB at SemEval-2020 Task 11: Propaganda Detection with Domain-Specific Trained BERT , 2020, SEMEVAL.

[48]  Victoria L. O'Donnell,et al.  Propaganda And Persuasion , 1989 .

[49]  Yann Mathet,et al.  The Unified and Holistic Method Gamma (γ) for Inter-Annotator Agreement Measure and Alignment , 2015, CL.

[50]  Aristides Gionis,et al.  Quantifying Controversy on Social Media , 2018, ACM Trans. Soc. Comput..

[51]  Daryna Dementieva,et al.  SkoltechNLP at SemEval-2020 Task 11: Exploring Unsupervised Text Augmentation for Propaganda Detection , 2020, SemEval@COLING.

[52]  Malak Abdullah,et al.  JUST at SemEval-2020 Task 11: Detecting Propaganda Techniques Using BERT Pre-trained Model , 2020, SemEval@COLING.

[53]  Preslav Nakov,et al.  CheckThat! at CLEF 2019: Automatic Identification and Verification of Claims , 2019, ECIR.

[54]  Preslav Nakov,et al.  Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection , 2019, EMNLP.

[55]  Guillaume Daval-Frerot,et al.  WMD at SemEval-2020 Tasks 7 and 11: Assessing Humor and Propaganda Using Unsupervised Data Augmentation , 2020, SEMEVAL.

[56]  Jiaxu Dao,et al.  YNU-HPCC at SemEval-2020 Task 11: LSTM Network for Detection of Propaganda Techniques in News Articles , 2020, SemEval@COLING.

[57]  Michael Kranzlein,et al.  Team DoNotDistribute at SemEval-2020 Task 11: Features, Finetuning, and Data Augmentation in Neural Models for Propaganda Detection in News Articles , 2020, SemEval@COLING.

[58]  Ekansh Verma,et al.  Transformers at SemEval-2020 Task 11: Propaganda Fragment Detection Using Diversified BERT Architectures Based Ensemble Learning , 2020, SEMEVAL.

[59]  Swati Agarwal,et al.  BPGC at SemEval-2020 Task 11: Propaganda Detection in News Articles with Multi-Granularity Knowledge Sharing and Linguistic Features based Ensemble Learning , 2020, SemEval@COLING.

[60]  Georgios Siolas,et al.  NTUAAILS at SemEval-2020 Task 11: Propaganda Detection and Classification with biLSTMs and ELMo , 2020, SemEval@COLING.

[61]  Preslav Nakov,et al.  CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media , 2020, ECIR.

[62]  Andreas Vlachos,et al.  FEVER: a Large-scale Dataset for Fact Extraction and VERification , 2018, NAACL.

[63]  Benno Stein,et al.  An Evaluation Framework for Plagiarism Detection , 2010, COLING.

[64]  Avi Arampatzis,et al.  DUTH at SemEval-2020 Task 11: BERT with Entity Mapping for Propaganda Classification , 2020, SemEval@COLING.

[65]  Preslav Nakov,et al.  Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness , 2018, CLEF.

[66]  Philip N. Howard,et al.  Computational Propaganda and Political Big Data: Moving Toward a More Critical Research Agenda , 2017, Big Data.

[67]  Suhang Wang,et al.  Fake News Detection on Social Media: A Data Mining Perspective , 2017, SKDD.

[68]  Robyn Torok.  Symbiotic radicalisation strategies: Propaganda tools and neuro linguistic programming , 2015 .

[69]  Alberto Barrón-Cedeño,et al.  On Automatic Plagiarism Detection Based on n-Grams Comparison , 2009, ECIR.

[70]  Indra Budi,et al.  3218IR at SemEval-2020 Task 11: Conv1D and Word Embedding in Propaganda Span Identification at News Articles , 2020, SEMEVAL.

[71]  Ryan L. Boyd,et al.  The Development and Psychometric Properties of LIWC2015 , 2015 .

[72]  Raymie McKerrow,et al.  Accounting for the force of the appeal to authority , 2011 .

[73]  Benno Stein,et al.  Computational Argumentation Quality Assessment in Natural Language , 2017, EACL.

[74]  Preslav Nakov,et al.  SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums , 2019, *SEMEVAL.

[75]  Barbara Plank,et al.  Team DiSaster at SemEval-2020 Task 11: Combining BERT and Hand-crafted Features for Identifying Propaganda Techniques in News , 2020, SEMEVAL.

[76]  Bo Zhao,et al.  A Survey on Truth Discovery , 2015, SKDD.

[77]  Miriam J. Metzger,et al.  The science of fake news , 2018, Science.

[78]  Stanley G. Robertson.  The Straw Man Fallacy , 2008 .