Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

The effectiveness of Neural Information Retrieval (Neu-IR) often depends on a large scale of in-domain relevance training signals, which are not always available in real-world ranking scenarios. To democratize the benefits of Neu-IR, this paper presents MetaAdaptRank, a domain adaptive learning method that generalizes Neu-IR models from label-rich source domains to few-shot target domains. Drawing on source-domain massive relevance supervision, MetaAdaptRank contrastively synthesizes a large number of weak supervision signals for target domains and meta-learns to reweight these synthetic “weak” data based on their benefits to the target-domain ranking accuracy of Neu-IR models. Experiments on three TREC benchmarks in the web, news, and biomedical domains show that MetaAdaptRank significantly improves the few-shot ranking accuracy of Neu-IR models. Further analyses indicate that MetaAdaptRank thrives from both its contrastive weak data synthesis and meta-reweighted data selection. The code and data of this paper can be obtained from https://github.com/thunlp/MetaAdaptRank.

[1]  Zhiyuan Liu,et al.  OpenMatch: An Open Source Library for Neu-IR Research , 2021, SIGIR.

[2]  Paul N. Bennett,et al.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval , 2020, ICLR.

[3]  Elizabeth Clark,et al.  Evaluation of Text Generation: A Survey , 2020, ArXiv.

[4]  Paul N. Bennett,et al.  Few-Shot Generative Conversational Query Rewriting , 2020, SIGIR.

[5]  W. Bruce Croft,et al.  Open-Retrieval Conversational Question Answering , 2020, SIGIR.

[6]  Allan Hanbury,et al.  Local Self-Attention over Long Text for Efficient Document Retrieval , 2020, SIGIR.

[7]  Kirk Roberts,et al.  TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19 , 2020, J. Am. Medical Informatics Assoc..

[8]  Oren Etzioni,et al.  CORD-19: The Covid-19 Open Research Dataset , 2020, NLPCOVID19.

[9]  Yulia Tsvetkov,et al.  Balancing Training for Multilingual Neural Machine Translation , 2020, ACL.

[10]  Jimmy J. Lin,et al.  Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset , 2020, NLPCOVID19.

[11]  Bhaskar Mitra,et al.  Overview of the TREC 2019 deep learning track , 2020, ArXiv.

[12]  Chenyan Xiong,et al.  Selective Weak Supervision for Neural Information Retrieval , 2020, WWW.

[13]  Jimmy J. Lin,et al.  Capreolus: A Toolkit for End-to-End Neural Ad Hoc Retrieval , 2020, WSDM.

[14]  Tom M. Mitchell,et al.  Learning Data Manipulation for Augmentation and Weighting , 2019, NeurIPS.

[15]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[16]  Chenyan Xiong,et al.  Fine-grained Fact Verification with Kernel Graph Attention Network , 2019, ACL.

[17]  Ahmed Hassan Awadallah,et al.  Meta Label Correction for Learning with Weak Supervision , 2019, ArXiv.

[18]  Yiqun Liu,et al.  Investigating Weak Supervision in Deep Ranking , 2019, Data Inf. Manag..

[19]  Guido Zuccon,et al.  TrecTools: an Open-source Python Library for Information Retrieval Practitioners Involved in TREC-like Campaigns , 2019, SIGIR.

[20]  Ming-Wei Chang,et al.  Latent Retrieval for Weakly Supervised Open Domain Question Answering , 2019, ACL.

[21]  Jamie Callan,et al.  Deeper Text Understanding for IR with Contextual Neural Language Modeling , 2019, SIGIR.

[22]  Jimmy J. Lin,et al.  Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models , 2019, SIGIR.

[23]  Zhiyuan Liu,et al.  Understanding the Behaviors of BERT in Ranking , 2019, ArXiv.

[24]  Qi Xie,et al.  Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting , 2019, NeurIPS.

[25]  Jimmy J. Lin,et al.  The Neural Hype and Comparisons Against Weak Baselines , 2019, SIGIR Forum.

[26]  Kyunghyun Cho,et al.  Passage Re-ranking with BERT , 2019, ArXiv.

[27]  Bhaskar Mitra,et al.  An Introduction to Neural Information Retrieval , 2018, Found. Trends Inf. Retr..

[28]  Gareth J. F. Jones,et al.  Challenges in the Development of Effective Systems for Professional Legal Search , 2018, ProfS/KG4IR/Data:Search@SIGIR.

[29]  Bin Yang,et al.  Learning to Reweight Examples for Robust Deep Learning , 2018, ICML.

[30]  Zhiyuan Liu,et al.  Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search , 2018, WSDM.

[31]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[32]  Jimmy J. Lin,et al.  Anserini: Enabling the Use of Lucene for Information Retrieval Research , 2017, SIGIR.

[33]  Andrew Yates,et al.  Content-Based Weak Supervision for Ad-Hoc Re-Ranking , 2017, SIGIR.

[34]  Tie-Yan Liu,et al.  Word-Entity Duet Representations for Document Ranking , 2017, SIGIR.

[35]  Zhiyuan Liu,et al.  End-to-End Neural Ad-hoc Ranking with Kernel Pooling , 2017, SIGIR.

[36]  W. Bruce Croft,et al.  Neural Ranking Models with Weak Supervision , 2017, SIGIR.

[37]  Gerard de Melo,et al.  PACRR: A Position-Aware Neural IR Model for Relevance Matching , 2017, EMNLP.

[38]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[39]  Jianfeng Gao,et al.  A Human Generated MAchine Reading COmprehension Dataset , 2018 .

[40]  W. Bruce Croft,et al.  A Deep Relevance Matching Model for Ad-hoc Retrieval , 2016, CIKM.

[41]  Hugo Zaragoza,et al.  The Probabilistic Relevance Framework: BM25 and Beyond , 2009, Found. Trends Inf. Retr..

[42]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[43]  Doug Downey,et al.  Heads and tails: studies of web search with common and rare queries , 2007, SIGIR.

[44]  W. Bruce Croft,et al.  Linear feature-based models for information retrieval , 2007, Information Retrieval.

[45]  W. Bruce Croft,et al.  A Markov random field model for term dependencies , 2005, SIGIR '05.

[46]  Wolfgang Nejdl,et al.  Using ODP metadata to personalize search , 2005, SIGIR '05.

[47]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[48]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[49]  Ryan T. McDonald,et al.  Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation , 2021, EACL.

[50]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[51]  Kui-Lam Kwok,et al.  TREC 2004 Robust Track Experiments Using PIRCS , 2004, TREC.

[52]  David Hawking,et al.  Challenges in Enterprise Search , 2004, ADC.