Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 1: Check-Worthiness

We present an overview of the 2nd edition of the CheckThat! Lab, part of CLEF 2019, with focus on Task 1: Check-Worthiness in political debates. The task asks to predict which claims in a political debate should be prioritized for fact-checking. In particular, given a debate or a political speech, the goal is to produce a ranked list of its sentences based on their worthiness for fact-checking. This year, we extended the 2018 dataset with 16 more debates and speeches. A total of 47 teams registered to participate in the lab, and eleven of them actually submitted runs for Task 1 (compared to seven last year). The evaluation results show that the most successful approaches to Task 1 used various neural networks and logistic regression. The best system achieved mean average precision of 0.166 (0.250 on the speeches, and 0.054 on the debates). This leaves large room for improvement, and thus we release all datasets and scoring scripts, which should enable further research in check-worthiness estimation.

[1]  Barbara Poblete,et al.  Information credibility on twitter , 2011, WWW.

[2]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[3]  Andreas Vlachos,et al.  Fact Checking: Task definition and dataset construction , 2014, LTCSS@ACL.

[4]  Carlos Angel Iglesias,et al.  Sematch: Semantic Entity Search from Knowledge Graph , 2015, SumPre-HSWI@ESWC.

[5]  Chengkai Li,et al.  Detecting Check-worthy Factual Claims in Presidential Debates , 2015, CIKM.

[6]  Naeemul Hassan,et al.  Comparing Automated Factual Claim Detection Against Judgments of Journalism Organizations , 2016 .

[7]  Filippo Menczer,et al.  Finding Streams in Knowledge Graphs to Support Fact Checking , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[8]  Saurabh Bagchi,et al.  TATHYA: A Multi-Classifier System for Detecting Check-Worthy Statements in Political Debates , 2017, CIKM.

[9]  Eunsol Choi,et al.  Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking , 2017, EMNLP.

[10]  Ganggao Zhu,et al.  Computing Semantic Similarity of Concepts in Knowledge Graphs , 2017, IEEE Transactions on Knowledge and Data Engineering.

[11]  Gerhard Weikum,et al.  Where the Truth Lies: Explaining the Credibility of Emerging Claims on the Web and Social Media , 2017, WWW.

[12]  Arkaitz Zubiaga,et al.  SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours , 2017, *SEMEVAL.

[13]  Preslav Nakov,et al.  Do Not Trust the Trolls: Predicting Credibility in Community Question Answering Forums , 2017, RANLP.

[14]  Preslav Nakov,et al.  Fully Automated Fact Checking Using External Sources , 2017, RANLP.

[15]  Preslav Nakov,et al.  A Context-Aware Approach for Detecting Worth-Checking Claims in Political Debates , 2017, RANLP.

[16]  Preslav Nakov,et al.  Integrating Stance Detection and Fact Checking in a Unified Corpus , 2018, NAACL.

[17]  Preslav Nakov,et al.  Fact Checking in Community Forums , 2018, AAAI.

[18]  Preslav Nakov,et al.  Predicting Factuality of Reporting and Bias of News Media Sources , 2018, EMNLP.

[19]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.

[20]  Preslav Nakov,et al.  ClaimRank: Detecting Check-Worthy Claims in Arabic and English , 2018, NAACL.

[21]  Preslav Nakov,et al.  Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness , 2018, CLEF.

[22]  Preslav Nakov,et al.  Automatic Stance Detection Using End-to-End Memory Networks , 2018, NAACL.

[23]  Nan Hua,et al.  Universal Sentence Encoder , 2018, ArXiv.

[24]  Ritwik Banerjee,et al.  A Hybrid Recognition System for Check-worthy Claims Using Heuristics and Supervised Learning , 2018, CLEF.

[25]  Christian Hansen,et al.  Neural Check-Worthiness Ranking with Weak Supervision: Finding Sentences for Fact-Checking , 2019, WWW.

[26]  Craig MacDonald,et al.  Entity Detection for Check-worthiness Prediction: Glasgow Terrier at CLEF CheckThat! 2019 , 2019, CLEF.

[27]  Dipankar Das,et al.  A Hybrid Model to Rank Sentences for Check-worthiness , 2019, CLEF.

[28]  Jakub Gasior,et al.  The IPIPAN Team Participation in the Check-Worthiness Task of the CLEF2019 CheckThat! Lab , 2019, CLEF.

[29]  Mucahid Kutlu,et al.  TOBB-ETU at CLEF 2019: Prioritizing Claims Based on Check-Worthiness , 2019, CLEF.

[30]  Pier Luca Lanzi,et al.  TheEarthIsFlat's Submission to CLEF'19CheckThat! Challenge , 2019, CLEF.

[31]  Preslav Nakov,et al.  CheckThat! at CLEF 2019: Automatic Identification and Verification of Claims , 2019, ECIR.

[32]  Christian Hansen,et al.  Neural Weakly Supervised Fact Check-Worthiness Detection with Contrastive Sampling-Based Ranking Loss , 2019, CLEF.

[33]  Preslav Nakov,et al.  Automatic Fact-Checking Using Context and Discourse Information , 2019, ACM J. Data Inf. Qual..

[34]  Preslav Nakov,et al.  SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums , 2019, *SEMEVAL.

[35]  Preslav Nakov,et al.  Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 2: Evidence and Factuality , 2019, CLEF.

[36]  Preslav Nakov,et al.  It Takes Nine to Smell a Rat: Neural Multi-Task Learning for Check-Worthiness Prediction , 2019, RANLP.

[37]  Giovanni Da San Martino,et al.  Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims , 2021, ArXiv.