Optimizing Test-Time Query Representations for Dense Retrieval

Recent developments of dense retrieval rely on quality representations of queries and contexts from pre-trained query and context encoders. In this paper, we introduce TOUR (Test-Time Optimization of Query Representations), which further optimizes instance-level query representations guided by signals from test-time retrieval results. We leverage a cross-encoder re-ranker to provide fine-grained pseudo labels over retrieval results and iteratively optimize query representations with gradient descent. Our theoretical analysis reveals that TOUR can be viewed as a generalization of the classical Rocchio algorithm for pseudo relevance feedback, and we present two variants that leverage pseudo-labels as hard binary or soft continuous labels. We first apply TOUR on phrase retrieval with our proposed phrase re-ranker, and also evaluate its effectiveness on passage retrieval with an off-the-shelf re-ranker. TOUR greatly improves end-to-end open-domain question answering accuracy, as well as passage retrieval performance. TOUR also consistently improves direct re-ranking by up to 2.0% while running 1.3-2.4x faster with an efficient implementation.

[1]  Rodrigo Nogueira,et al.  In Defense of Cross-Encoders for Zero-Shot Retrieval , 2022, ArXiv.

[2]  Leonardo F. R. Ribeiro,et al.  Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking , 2022, EMNLP.

[3]  Wenhan Xiong,et al.  Boosted Dense Retriever , 2021, NAACL.

[4]  Omer Levy,et al.  Learning to Retrieve Passages without Supervision , 2021, NAACL.

[5]  Wayne Xin Zhao,et al.  RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking , 2021, EMNLP.

[6]  Danqi Chen,et al.  Simple Entity-Centric Questions Challenge Dense Retrievers , 2021, EMNLP.

[7]  Danqi Chen,et al.  Phrase Retrieval Learns Passage Retrieval, Too , 2021, EMNLP.

[8]  Pavel Smrz,et al.  R2-D2: A Modular Baseline for Open-Domain Question Answering , 2021, EMNLP.

[9]  Chenyan Xiong,et al.  Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback , 2021, CIKM.

[10]  Hang Li,et al.  Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls , 2021, ACM Trans. Inf. Syst..

[11]  Iadh Ounis,et al.  Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval , 2021, ICTIR.

[12]  Dani Yogatama,et al.  End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering , 2021, NeurIPS.

[13]  Yashar Mehdad,et al.  RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering , 2021, NAACL.

[14]  Iryna Gurevych,et al.  BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models , 2021, NeurIPS Datasets and Benchmarks.

[15]  Jimmy J. Lin,et al.  Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling , 2021, SIGIR.

[16]  Yuxiang Wu,et al.  PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them , 2021, Transactions of the Association for Computational Linguistics.

[17]  Yelong Shen,et al.  UnitedQA: A Hybrid Approach for Open Domain Question Answering , 2021, ACL.

[18]  Danqi Chen,et al.  Learning Dense Representations of Phrases at Scale , 2020, ACL.

[19]  Edouard Grave,et al.  Distilling Knowledge from Reader to Retriever for Question Answering , 2020, ICLR.

[20]  Yelong Shen,et al.  Generation-Augmented Retrieval for Open-Domain Question Answering , 2020, ACL.

[21]  Sebastian Riedel,et al.  Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets , 2020, EACL.

[22]  Ming-Wei Chang,et al.  Retrieval Augmented Language Model Pre-Training , 2020, ICML.

[23]  Edouard Grave,et al.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering , 2020, EACL.

[24]  Christopher Potts,et al.  Relevance-guided Supervision for OpenQA with ColBERT , 2020, Transactions of the Association for Computational Linguistics.

[25]  Paul N. Bennett,et al.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval , 2020, ICLR.

[26]  M. Zaharia,et al.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT , 2020, SIGIR.

[27]  Danqi Chen,et al.  Dense Passage Retrieval for Open-Domain Question Answering , 2020, EMNLP.

[28]  Ming-Wei Chang,et al.  REALM: Retrieval-Augmented Language Model Pre-Training , 2020, ICML.

[29]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[30]  Ming-Wei Chang,et al.  Natural Questions: A Benchmark for Question Answering Research , 2019, TACL.

[31]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[32]  Ali Farhadi,et al.  Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index , 2019, ACL.

[33]  Ming-Wei Chang,et al.  Latent Retrieval for Weakly Supervised Open Domain Question Answering , 2019, ACL.

[34]  Yejin Choi,et al.  The Curious Case of Neural Text Degeneration , 2019, ICLR.

[35]  Steve Renals,et al.  Dynamic Evaluation of Transformer Language Models , 2019, ArXiv.

[36]  Kyunghyun Cho,et al.  Passage Re-ranking with BERT , 2019, ArXiv.

[37]  Ben He,et al.  NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval , 2018, EMNLP.

[38]  W. Bruce Croft,et al.  From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing , 2018, CIKM.

[39]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[40]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[41]  Petr Baudis,et al.  Modeling of the Question Answering Task in the YodaQA System , 2015, CLEF.

[42]  Andrew Chou,et al.  Semantic Parsing on Freebase from Question-Answer Pairs , 2013, EMNLP.

[43]  W. Bruce Croft,et al.  Relevance-Based Language Models , 2001, SIGIR '01.

[44]  W. Bruce Croft,et al.  Search Engines - Information Retrieval in Practice , 2009 .

[45]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .