BERT-QPP: Contextualized Pre-trained transformers for Query Performance Prediction

Query Performance Prediction (QPP) is focused on estimating the difficulty of satisfying a user query for a certain retrieval method. While most state of the art QPP methods are based on term frequency and corpus statistics, more recent work in this area have started to explore the utility of pretrained neural embeddings, neural architectures and contextual embeddings. Such approaches extract features from pretrained or contextual embeddings for the sake of training a supervised performance predictor. In this paper, we adopt contextual embeddings to perform performance prediction, but distinguish ourselves from the state of the art by proposing to directly fine-tune a contextual embedding, i.e., BERT, specifically for the task of query performance prediction. As such, our work allows the fine-tuned contextual representations to estimate the performance of a query based on the association between the representation of the query and the retrieved documents. We compare the performance of our approach with the state-of-the-art based on the MS MARCO passage retrieval corpus and its three associated query sets: (1) MS MARCO development set, (2) TREC DL 2019, and (3) TREC DL 2020. We show that our approach not only shows significant improved prediction performance compared to all the state-of-the-art methods, but also, unlike past neural predictors, it shows significantly lower latency, making it possible to use in practice.

[1]  W. Bruce Croft,et al.  Performance Prediction for Non-Factoid Question Answering , 2019, ICTIR.

[2]  Danqi Chen,et al.  Dense Passage Retrieval for Open-Domain Question Answering , 2020, EMNLP.

[3]  Claudia Hauff,et al.  Predicting the effectiveness of queries and retrieval systems , 2010, SIGF.

[4]  Jelena Jovanovic,et al.  Geometric Estimation of Specificity within Embedding Spaces , 2019, CIKM.

[5]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[6]  Ye Li,et al.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval , 2020, ArXiv.

[7]  Haggai Roitman,et al.  ICTIR Tutorial: Modern Query Performance Prediction: Theory and Practice , 2020, ICTIR.

[8]  Jason Weston,et al.  Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring , 2019 .

[9]  Guy Feigenblat,et al.  A Study of Query Performance Prediction for Answer Quality Determination , 2019, ICTIR.

[10]  Thomas Wolf,et al.  TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents , 2019, ArXiv.

[11]  Ebrahim Bagheri,et al.  Semantics-enabled query performance prediction for ad hoc table retrieval , 2021, Inf. Process. Manag..

[12]  Haggai Roitman,et al.  Robust Standard Deviation Estimation for Query Performance Prediction , 2017, ICTIR.

[13]  Oren Kurland,et al.  Query-performance prediction: setting the expectations straight , 2014, SIGIR.

[14]  Elad Yom-Tov,et al.  Estimating the query difficulty for information retrieval , 2010, Synthesis Lectures on Information Concepts, Retrieval, and Services.

[15]  Bhaskar Mitra,et al.  An Introduction to Neural Information Retrieval , 2018, Found. Trends Inf. Retr..

[16]  Jimmy J. Lin,et al.  Distilling Dense Representations for Ranking using Tightly-Coupled Teachers , 2020, ArXiv.

[17]  Oren Kurland,et al.  Using statistical decision theory and relevance models for query-performance prediction , 2010, SIGIR.

[18]  Jelena Jovanovic,et al.  Neural embedding-based specificity metrics for pre-retrieval query performance prediction , 2020, Inf. Process. Manag..

[19]  Joemon M. Jose,et al.  Improved query performance prediction using standard deviation , 2011, SIGIR.

[20]  Stephen E. Robertson,et al.  A probabilistic model of information retrieval: development and comparative experiments - Part 2 , 2000, Inf. Process. Manag..

[21]  Jianfeng Gao,et al.  A Human Generated MAchine Reading COmprehension Dataset , 2018 .

[22]  Lourdes Araujo,et al.  Standard Deviation as a Query Hardness Estimator , 2010, SPIRE.

[23]  J. Shane Culpepper,et al.  Neural Query Performance Prediction using Weak Supervision from Multiple Signals , 2018, SIGIR.

[24]  Yiqun Liu,et al.  RepBERT: Contextualized Text Embeddings for First-Stage Retrieval , 2020, ArXiv.

[25]  M. Zaharia,et al.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT , 2020, SIGIR.

[26]  Gary Marchionini,et al.  Synthesis Lectures on Information Concepts, Retrieval, and Services , 2009 .

[27]  Iryna Gurevych,et al.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks , 2019, EMNLP.

[28]  W. Bruce Croft,et al.  Relevance-Based Language Models , 2001, SIGIR '01.

[29]  Azadeh Shakery,et al.  Deep Neural Networks for Query Expansion using Word Embeddings , 2018, ECIR.

[30]  Jelena Jovanovic,et al.  Neural Embedding-Based Metrics for Pre-retrieval Query Performance Prediction , 2020, ECIR.

[31]  W. Bruce Croft,et al.  Query performance prediction in web search environments , 2007, SIGIR.

[32]  Oren Kurland,et al.  Predicting Query Performance by Query-Drift Estimation , 2009, TOIS.

[33]  Hua Wu,et al.  RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering , 2020, NAACL.

[34]  Shengli Wu,et al.  Query Performance Prediction By Considering Score Magnitude and Variance Together , 2014, CIKM.

[35]  Gareth J. F. Jones,et al.  Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction , 2019, Inf. Process. Manag..

[36]  M. Zaharia,et al.  ColBERT , 2020, Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.

[37]  Ebrahim Bagheri,et al.  Query Performance Prediction Through Retrieval Coherency , 2021, ECIR.

[38]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[39]  J. Leon Zhao,et al.  Query‐performance prediction for effective query routing in domain‐specific repositories , 2014, J. Assoc. Inf. Sci. Technol..

[40]  Bhaskar Mitra,et al.  Overview of the TREC 2019 deep learning track , 2020, ArXiv.

[41]  Kyunghyun Cho,et al.  Passage Re-ranking with BERT , 2019, ArXiv.

[42]  Jesse Vig,et al.  Comparison of Transfer-Learning Approaches for Response Selection in Multi-Turn Conversations , 2018 .