DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking

The increased focus on misinformation has spurred development of data and systems for detecting the veracity of a claim as well as retrieving authoritative evidence. The Fact Extraction and VERification (FEVER) dataset provides such a resource for evaluating end-to-end fact-checking, requiring retrieval of evidence from Wikipedia to validate a veracity prediction. We show that current systems for FEVER are vulnerable to three categories of realistic challenges for fact-checking -- multiple propositions, temporal reasoning, and ambiguity and lexical variation -- and introduce a resource with these types of claims. Then we present a system designed to be resilient to these "attacks" using multiple pointer networks for document selection and jointly modeling a sequence of evidence sentences and veracity relation predictions. We find that in handling these attacks we obtain state-of-the-art results on FEVER, largely due to improved evidence retrieval.

[1]  Samy Bengio,et al.  Order Matters: Sequence to sequence for sets , 2015, ICLR.

[2]  Smaranda Muresan,et al.  Robust Document Retrieval and Individual Evidence Modeling for Fact Extraction and Verification. , 2018, FEVER@EMNLP.

[3]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[4]  William Yang Wang.  “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection , 2017, ACL.

[5]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[6]  Preslav Nakov,et al.  Fact Checking in Community Forums , 2018, AAAI.

[7]  Andreas Vlachos,et al.  Adversarial attacks against Fact Extraction and VERification , 2019, ArXiv.

[8]  Sebastian Riedel,et al.  UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF) , 2018, FEVER@EMNLP.

[9]  Gabriel Stanovsky,et al.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs , 2019, NAACL.

[10]  Niloy Ganguly,et al.  AttentiveChecker: A Bi-Directional Attention Flow Mechanism for Fact Verification , 2019, NAACL.

[11]  Percy Liang,et al.  Adversarial Examples for Evaluating Reading Comprehension Systems , 2017, EMNLP.

[12]  Gordon Korman,et al.  Liar, Liar, Pants on Fire , 1997 .

[13]  Preslav Nakov,et al.  SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums , 2019, *SEMEVAL.

[14]  Mona T. Diab,et al.  Team SWEEPer: Joint Sentence Extraction and Fact Checking with Pointer Networks , 2018 .

[15]  Sinan Aral,et al.  The spread of true and false news online , 2018, Science.

[16]  Haonan Chen,et al.  Combining Fact Extraction and Verification with Neural Semantic Matching Networks , 2018, AAAI.

[17]  Dan Roth,et al.  TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification , 2018, EMNLP.

[18]  Andreas Vlachos,et al.  Fact Checking: Task definition and dataset construction , 2014, LTCSS@ACL.

[19]  Jason Weston,et al.  End-To-End Memory Networks , 2015, NIPS.

[20]  Eunsol Choi,et al.  Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking , 2017, EMNLP.

[21]  Dominik Stammbach,et al.  Team DOMLIN: Exploiting Evidence Enhancement for the FEVER Shared Task , 2019, EMNLP.

[22]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[23]  Andreas Vlachos,et al.  An Extensible Framework for Verification of Numerical Claims , 2017, EACL.

[24]  Kenton Lee,et al.  Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension , 2019, EMNLP.

[25]  Mani B. Srivastava,et al.  Generating Natural Language Adversarial Examples , 2018, EMNLP.

[26]  Maosong Sun,et al.  GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification , 2019, ACL.

[27]  Christopher Malon,et al.  Team Papelo: Transformer Networks at FEVER , 2019, ArXiv.

[28]  Sameer Singh,et al.  Universal Adversarial Triggers for Attacking and Analyzing NLP , 2019, EMNLP.

[29]  Navdeep Jaitly,et al.  Pointer Networks , 2015, NIPS.

[30]  Christopher Potts,et al.  A large annotated corpus for learning natural language inference , 2015, EMNLP.

[31]  Lucas Graves.  Understanding the Promise and Limits of Automated Fact-Checking , 2018 .

[32]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[33]  Ido Dagan,et al.  Supervised Open Information Extraction , 2018, NAACL.

[34]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[35]  Mohit Bansal,et al.  Adversarial NLI: A New Benchmark for Natural Language Understanding , 2020, ACL.

[36]  Tom M. Mitchell,et al.  Language-Aware Truth Assessment of Fact Candidates , 2014, ACL.

[37]  Paramita Mirza,et al.  CATENA: CAusal and TEmporal relation extraction from NAtural language texts , 2016, COLING.

[38]  Claire Cardie,et al.  Identifying Appropriate Support for Propositions in Online User Comments , 2014, ArgMining@ACL.

[39]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[40]  Masaaki Nagata,et al.  Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction , 2019, ACL.

[41]  Yen-Chun Chen,et al.  Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting , 2018, ACL.

[42]  Smaranda Muresan,et al.  Where is Your Evidence: Improving Fact-checking by Justification Modeling , 2018 .

[43]  Iryna Gurevych,et al.  UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification , 2018, FEVER@EMNLP.

[44]  Wanxiang Che,et al.  Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency , 2019, ACL.

[45]  Andreas Vlachos,et al.  FEVER: a Large-scale Dataset for Fact Extraction and VERification , 2018, NAACL.

[46]  Yoav Goldberg,et al.  Breaking NLI Systems with Sentences that Require Simple Lexical Inferences , 2018, ACL.

[47]  Christo Wilson,et al.  Linguistic Signals under Misinformation and Fact-Checking , 2018, Proc. ACM Hum. Comput. Interact..

[48]  James Allan,et al.  FEVER Breaker’s Run of Team NbAuzDrLqg , 2019, EMNLP.

[49]  Chengkai Li,et al.  Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster , 2017, KDD.

[50]  Kathy McKeown,et al.  Identifying Causal Relations Using Parallel Wikipedia Articles , 2016, ACL.

[51]  Andreas Vlachos,et al.  Automated Fact Checking: Task Formulations, Methods and Future Directions , 2018, COLING.

[52]  Carlos Guestrin,et al.  Semantically Equivalent Adversarial Rules for Debugging NLP models , 2018, ACL.