CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media

We describe the third edition of the CheckThat! Lab, which is part of the 2020 Cross-Language Evaluation Forum (CLEF). CheckThat! proposes four complementary tasks and a related task from previous lab editions, offered in English, Arabic, and Spanish. Task 1 asks to predict which tweets in a Twitter stream are worth fact-checking. Task 2 asks to determine whether a claim posted in a tweet can be verified using a set of previously fact-checked claims. Task 3 asks to retrieve text snippets from a given set of Web pages that would be useful for verifying a target tweet’s claim. Task 4 asks to predict the veracity of a target tweet’s claim using a set of potentially-relevant Web pages. Finally, the lab offers a fifth task that asks to predict the check-worthiness of the claims made in English political debates and speeches. CheckThat! features a full evaluation framework. The evaluation is carried out using mean average precision or precision at rank k for ranking tasks, and F\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_1$$\end{document} for classification tasks.

[1]  Alessandro Moschitti,et al.  Structural Representations for Learning Relations between Pairs of Texts , 2015, ACL.

[2]  Preslav Nakov,et al.  Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 2: Evidence and Factuality , 2019, CLEF.

[3]  Tamer Elsayed,et al.  bigIR at CLEF 2019: Automatic Verification of Arabic Claims over the Web , 2019, CLEF.

[4]  Wei Gao,et al.  Detecting Rumors from Microblogs with Recurrent Neural Networks , 2016, IJCAI.

[5]  Gerhard Weikum,et al.  Credibility Assessment of Textual Claims on the Web , 2016, CIKM.

[6]  Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018 , 2018, CLEF.

[7]  Preslav Nakov,et al.  Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 1: Check-Worthiness , 2019, CLEF.

[8]  Eric Gilbert,et al.  CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations , 2015, ICWSM.

[9]  Jakub Gasior,et al.  The IPIPAN Team Participation in the Check-Worthiness Task of the CLEF2019 CheckThat! Lab , 2019, CLEF.

[10]  Preslav Nakov,et al.  A Context-Aware Approach for Detecting Worth-Checking Claims in Political Debates , 2017, RANLP.

[11]  Jakob Grue Simonsen,et al.  The Copenhagen Team Participation in the Check-Worthiness Task of the Competition of Automatic Identification and Verification of Claims in Political Debates of the CLEF-2018 CheckThat! Lab , 2018, CLEF.

[12]  Preslav Nakov,et al.  It Takes Nine to Smell a Rat: Neural Multi-Task Learning for Check-Worthiness Prediction , 2019, RANLP.

[13]  Ioana Manolescu,et al.  A Content Management Perspective on Fact-Checking , 2018, WWW.

[14]  Eneko Agirre,et al.  SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation , 2017, *SEMEVAL.

[15]  Chengkai Li,et al.  ClaimBuster: The First-ever End-to-end Fact-checking System , 2017, Proc. VLDB Endow..

[16]  Qiaozhu Mei,et al.  Enquiring Minds: Early Detection of Rumors in Social Media from Enquiry Posts , 2015, WWW.

[17]  Ritwik Banerjee,et al.  A Hybrid Recognition System for Check-worthy Claims Using Heuristics and Supervised Learning , 2018, CLEF.

[18]  Paolo Rosso,et al.  UPV-INAOE - Check That: Preliminary Approach for Checking Worthiness of Claims , 2018, CLEF.

[19]  Preslav Nakov,et al.  Fully Automated Fact Checking Using External Sources , 2017, RANLP.

[20]  Gerhard Weikum,et al.  Leveraging Joint Interactions for Credibility Analysis in News Communities , 2015, CIKM.

[21]  Preslav Nakov,et al.  Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness , 2018, CLEF.

[22]  Pier Luca Lanzi,et al.  TheEarthIsFlat's Submission to CLEF'19CheckThat! Challenge , 2019, CLEF.

[23]  Preslav Nakov,et al.  SemEval-2016 Task 3: Community Question Answering , 2019, *SEMEVAL.

[24]  Laure Berti-Équille,et al.  VERA: A Platform for Veracity Estimation over Web Data , 2016, WWW.

[25]  Preslav Nakov,et al.  CheckThat! at CLEF 2019: Automatic Identification and Verification of Claims , 2019, ECIR.

[26]  Preslav Nakov,et al.  Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 2: Factuality , 2018, CLEF.

[27]  Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019 , 2019, CLEF.

[28]  Suhang Wang,et al.  Fake News Detection on Social Media: A Data Mining Perspective , 2017, SKDD.

[29]  Arkaitz Zubiaga,et al.  Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads , 2015, PloS one.

[30]  Naeemul Hassan,et al.  Comparing Automated Factual Claim Detection Against Judgments of Journalism Organizations , 2016 .

[31]  Christian Hansen,et al.  Neural Weakly Supervised Fact Check-Worthiness Detection with Contrastive Sampling-Based Ranking Loss , 2019, CLEF.

[32]  Ponnurangam Kumaraguru,et al.  TweetCred: Real-Time Credibility Assessment of Content on Twitter , 2014, SocInfo.

[33]  Chengkai Li,et al.  Detecting Check-worthy Factual Claims in Presidential Debates , 2015, CIKM.

[34]  P. Glasziou,et al.  Rural Doctors’ Views on and Experiences with Evidence-Based Medicine: The FrEEDoM Qualitative Study , 2016, PloS one.

[35]  Paolo Rosso,et al.  UPV-UMA at CheckThat! Lab: Verifying Arabic Claims using a Cross Lingual Approach , 2019, CLEF.