Exploring Listwise Evidence Reasoning with T5 for Fact Verification

This work explores a framework for fact verification that leverages pretrained sequence-to-sequence transformer models for sentence selection and label prediction, two key sub-tasks in fact verification. Most notably, improving on previous pointwise aggregation approaches for label prediction, we take advantage of T5 using a listwise approach coupled with data augmentation. With this enhancement, we observe that our label prediction stage is more robust to noise and capable of verifying complex claims by jointly reasoning over multiple pieces of evidence. Experimental results on the FEVER task show that our system attains a FEVER score of 75.87% on the blind test set. This puts our approach atop the competitive FEVER leaderboard at the time of our work, scoring higher than the second place submission by almost two points in label accuracy and over one point in FEVER score.

[1]  Iryna Gurevych,et al.  UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification , 2018, FEVER@EMNLP.

[2]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[3]  Jimmy J. Lin,et al.  Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval , 2019, EMNLP.

[4]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[5]  Marcel Worring,et al.  BERT for Evidence Retrieval and Claim Verification , 2019, ECIR.

[6]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[7]  Hugo Zaragoza,et al.  The Probabilistic Relevance Framework: BM25 and Beyond , 2009, Found. Trends Inf. Retr..

[8]  Sebastian Riedel,et al.  UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF) , 2018, FEVER@EMNLP.

[9]  Jimmy J. Lin,et al.  H2oloo at TREC 2020: When all you got is a hammer... Deep Learning, Health Misinformation, and Precision Medicine , 2020, TREC.

[10]  Mohit Bansal,et al.  Revealing the Importance of Semantic Retrieval for Machine Reading at Scale , 2019, EMNLP.

[11]  Zhenghao Liu,et al.  Coreferential Reasoning Learning for Language Representation , 2020, EMNLP.

[12]  Jimmy J. Lin,et al.  Anserini: Enabling the Use of Lucene for Information Retrieval Research , 2017, SIGIR.

[13]  Jimmy J. Lin,et al.  Vera: Prediction Techniques for Reducing Harmful Misinformation in Consumer Health Search , 2021, SIGIR.

[14]  Haonan Chen,et al.  Combining Fact Extraction and Verification with Neural Semantic Matching Networks , 2018, AAAI.

[15]  M. Zhou,et al.  Reasoning Over Semantic-Level Graph for Fact Checking , 2019, ACL.

[16]  Zhen-Hua Ling,et al.  Enhanced LSTM for Natural Language Inference , 2016, ACL.

[17]  Shyam Subramanian,et al.  Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification , 2020, EMNLP.

[18]  Wei Zhang,et al.  Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering , 2017, ICLR.

[19]  Chenyan Xiong,et al.  Fine-grained Fact Verification with Kernel Graph Attention Network , 2019, ACL.

[20]  Andreas Vlachos,et al.  FEVER: a Large-scale Dataset for Fact Extraction and VERification , 2018, NAACL.

[21]  Maosong Sun,et al.  GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification , 2019, ACL.

[22]  Jimmy J. Lin,et al.  Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations , 2021, SIGIR.

[23]  Jimmy J. Lin,et al.  The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models , 2021, ArXiv.

[24]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[25]  Jimmy J. Lin,et al.  Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset , 2020, SDP.

[26]  Jimmy J. Lin,et al.  Scientific Claim Verification with VerT5erini , 2020, LOUHI.

[27]  Dominik Stammbach,et al.  Team DOMLIN: Exploiting Evidence Enhancement for the FEVER Shared Task , 2019, EMNLP.

[28]  Jimmy J. Lin,et al.  Document Ranking with a Pretrained Sequence-to-Sequence Model , 2020, FINDINGS.

[29]  Smaranda Muresan,et al.  Robust Document Retrieval and Individual Evidence Modeling for Fact Extraction and Verification. , 2018, FEVER@EMNLP.