Scientific Claim Verification with VerT5erini

This work describes the adaptation of a pretrained sequence-to-sequence model to the task of scientific claim verification in the biomedical domain. We propose VERT5ERINI that exploits T5 for abstract retrieval, sentence selection and label prediction, which are three critical sub-tasks of claim verification. We evaluate our pipeline on SCIFACT, a newly curated dataset that requires models to not just predict the veracity of claims but also provide relevant sentences from a corpus of scientific literature that support this decision. Empirically, our pipeline outperforms a strong baseline in each of the three steps. Finally, we show VERT5ERINI's ability to generalize to two new datasets of COVID-19 claims using evidence from the ever-expanding CORD-19 corpus.

[1]  Jimmy J. Lin,et al.  Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset , 2020, SDP.

[2]  Jimmy J. Lin,et al.  Rapidly Bootstrapping a Question Answering Dataset for COVID-19 , 2020, ArXiv.

[3]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[4]  Jimmy J. Lin,et al.  Document Ranking with a Pretrained Sequence-to-Sequence Model , 2020, FINDINGS.

[5]  Nazli Goharian,et al.  CEDR: Contextualized Embeddings for Document Ranking , 2019, SIGIR.

[6]  Jimmy J. Lin,et al.  Anserini: Enabling the Use of Lucene for Information Retrieval Research , 2017, SIGIR.

[7]  Byron C. Wallace,et al.  ERASER: A Benchmark to Evaluate Rationalized NLP Models , 2020, ACL.

[8]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[9]  Kirk Roberts,et al.  TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19 , 2020, J. Am. Medical Informatics Assoc..

[10]  Iryna Gurevych,et al.  A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking , 2019, CoNLL.

[11]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[12]  Hang Li.  Learning to Rank for Information Retrieval and Natural Language Processing , 2011, Synthesis Lectures on Human Language Technologies.

[13]  Jimmy J. Lin,et al.  Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval , 2019, EMNLP.

[14]  Zhuyun Dai,et al.  Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval , 2019, ArXiv.

[15]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[16]  Stephen E. Robertson,et al.  GatfordCentre for Interactive Systems ResearchDepartment of Information , 1996 .

[17]  Jimmy J. Lin,et al.  Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations , 2021, ArXiv.

[18]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[19]  Andreas Vlachos,et al.  FEVER: a Large-scale Dataset for Fact Extraction and VERification , 2018, NAACL.

[20]  Arman Cohan,et al.  SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search , 2020, ArXiv.

[21]  Jimmy J. Lin,et al.  Document Expansion by Query Prediction , 2019, ArXiv.

[22]  William Yang Wang.  “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection , 2017, ACL.

[23]  Jimmy J. Lin,et al.  The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models , 2021, ArXiv.

[24]  Marcel Worring,et al.  BERT for Evidence Retrieval and Claim Verification , 2019, ECIR.

[25]  Charles L. A. Clarke,et al.  A Lightweight Environment for Learning Experimental IR Research Practices , 2020, SIGIR.

[26]  Hannaneh Hajishirzi,et al.  Fact or Fiction: Verifying Scientific Claims , 2020, EMNLP.

[27]  Samuel R. Bowman,et al.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference , 2017, NAACL.

[28]  Nayeon Lee,et al.  Misinformation Has High Perplexity , 2020, ArXiv.

[29]  Jianfeng Gao,et al.  A Human Generated MAchine Reading COmprehension Dataset , 2018 .