A Deep Look into Neural Ranking Models for Information Retrieval

Abstract Ranking models lie at the heart of research on information retrieval (IR). During the past decades, different techniques have been proposed for constructing ranking models, from traditional heuristic methods, probabilistic methods, to modern machine learning methods. Recently, with the advance of deep learning technology, we have witnessed a growing body of work in applying shallow or deep neural networks to the ranking problem in IR, referred to as neural ranking models in this paper. The power of neural ranking models lies in the ability to learn from the raw text inputs for the ranking problem to avoid many limitations of hand-crafted features. Neural networks have sufficient capacity to model complicated tasks, which is needed to handle the complexity of relevance estimation in ranking. Since there have been a large variety of neural ranking models proposed, we believe it is the right time to summarize the current status, learn from existing methodologies, and gain some insights for future development. In contrast to existing reviews, in this survey, we will take a deep look into the neural ranking models from different dimensions to analyze their underlying assumptions, major design principles, and learning strategies. We compare these models through benchmark tasks to obtain a comprehensive empirical understanding of the existing techniques. We will also discuss what is missing in the current literature and what are the promising and desired future directions.

[1]  Matthew Richardson,et al.  MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text , 2013, EMNLP.

[2]  Emine Yilmaz,et al.  Semi-supervised learning to rank with preference regularization , 2011, CIKM '11.

[3]  Bhaskar Mitra,et al.  A Proposal for Evaluating Answer Distillation from Web Data , 2016 .

[4]  Preslav Nakov,et al.  SIGIR 2016 Workshop WebQA II: Web Question Answering Beyond Factoids , 2016, SIGIR.

[5]  Gerard de Melo,et al.  Co-PACRR: A Context-Aware Neural IR Model for Ad-hoc Retrieval , 2017, WSDM.

[6]  Zhiyuan Liu,et al.  End-to-End Neural Ad-hoc Ranking with Kernel Pooling , 2017, SIGIR.

[7]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[8]  Hang Li,et al.  Convolutional Neural Network Architectures for Matching Natural Language Sentences , 2014, NIPS.

[9]  Zhen Xu,et al.  Incorporating Loose-Structured Knowledge into LSTM with Recall Gate for Conversation Modeling , 2016, ArXiv.

[10]  W. Bruce Croft,et al.  Adaptability of Neural Networks on Varying Granularity IR Tasks , 2016, ArXiv.

[11]  Meng Zhang,et al.  Neural Network Methods for Natural Language Processing , 2017, Computational Linguistics.

[12]  Jaap Kamps,et al.  Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision , 2017, ArXiv.

[13]  Dong Liu,et al.  MIX: Multi-Channel Information Crossing for Text Matching , 2018, KDD.

[14]  Huiping Sun,et al.  CQArank: jointly model topics and expertise in community question answering , 2013, CIKM.

[15]  Joelle Pineau,et al.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems , 2015, SIGDIAL Conference.

[16]  W. Bruce Croft,et al.  A Deep Relevance Matching Model for Ad-hoc Retrieval , 2016, CIKM.

[17]  Preslav Nakov,et al.  SemEval-2017 Task 3: Community Question Answering , 2017, *SEMEVAL.

[18]  Jianfeng Gao,et al.  A Human Generated MAchine Reading COmprehension Dataset , 2018 .

[19]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[20]  Maarten de Rijke,et al.  ViTOR: Learning to Rank Webpages Based on Visual Features , 2019, WWW.

[21]  Bo Li,et al.  Joint Learning from Labeled and Unlabeled Data for Information Retrieval , 2018, COLING.

[22]  Marti A. Hearst.  TileBars: visualization of term distribution information in full text information access , 1995, CHI '95.

[23]  Xuan Liu,et al.  Multi-view Response Selection for Human-Computer Conversation , 2016, EMNLP.

[24]  Jian-Yun Nie,et al.  Empirical Study of Multi-level Convolution Models for IR Based on Representations and Interactions , 2018, ICTIR.

[25]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[26]  Tim Kraska,et al.  The Case for Learned Index Structures , 2018 .

[27]  W. Bruce Croft,et al.  Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation , 2017, ArXiv.

[28]  Yiqun Liu,et al.  Sogou-QCL: A New Dataset with Click Relevance Label , 2018, SIGIR.

[29]  Yiqun Liu,et al.  Unbiased Learning to Rank: Theory and Practice , 2018, ICTIR.

[30]  M. de Rijke,et al.  Attention-based Hierarchical Neural Query Suggestion , 2018, SIGIR.

[31]  Jianfeng Gao,et al.  Neural Approaches to Conversational AI , 2018, ACL.

[32]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[33]  W. Bruce Croft,et al.  Analyzing and Characterizing User Intent in Information-seeking Conversations , 2018, SIGIR.

[34]  Xueqi Cheng,et al.  A Study of MatchPyramid Models on Ad-hoc Retrieval , 2016, ArXiv.

[35]  W. Bruce Croft,et al.  A Language Modeling Approach to Information Retrieval , 1998, SIGIR Forum.

[36]  Gerhard Weikum,et al.  ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters , 2018, NAACL.

[37]  Xueqi Cheng,et al.  MatchZoo: A Toolkit for Deep Text Matching , 2017, ArXiv.

[38]  W. Bruce Croft,et al.  Relevance-Based Language Models , 2001, SIGIR '01.

[39]  Zhoujun Li,et al.  Knowledge Enhanced Hybrid Neural Network for Text Matching , 2018, AAAI.

[40]  Wenpeng Yin,et al.  MultiGranCNN: An Architecture for General Matching of Text Chunks on Multiple Levels of Granularity , 2015, ACL.

[41]  W. Bruce Croft,et al.  Neural Ranking Models with Weak Supervision , 2017, SIGIR.

[42]  David Novak,et al.  Off the Beaten Path: Let's Replace Term-Based Retrieval with k-NN Search , 2016, CIKM.

[43]  Filip Radlinski,et al.  TREC Complex Answer Retrieval Overview , 2018, TREC.

[44]  Filip Radlinski,et al.  Inferring and using location metadata to personalize web search , 2011, SIGIR.

[45]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[46]  Xueqi Cheng,et al.  DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval , 2017, CIKM.

[47]  Ming-Wei Chang,et al.  Question Answering Using Enhanced Lexical Semantic Models , 2013, ACL.

[48]  Jun Zhao,et al.  End-to-End Neural Ranking for eCommerce Product Search: an Application of Task Models and Textual Embeddings , 2018, eCOM@SIGIR.

[49]  W. Bruce Croft,et al.  Beyond Factoid QA: Effective Methods for Non-factoid Answer Sentence Retrieval , 2016, ECIR.

[50]  ChengXiang Zhai,et al.  LinkSO: a dataset for learning to retrieve similar question answer pairs on software development forums , 2018, NL4SE@ESEC/SIGSOFT FSE.

[51]  Nick Craswell,et al.  Learning to Match using Local and Distributed Representations of Text for Web Search , 2016, WWW.

[52]  Sebastian Bruch,et al.  Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks , 2018, ICTIR.

[53]  Ellen M. Voorhees,et al.  Building a question answering test collection , 2000, SIGIR '00.

[54]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[56]  Xueqi Cheng,et al.  Learning Visual Features from Snapshots for Web Search , 2017, CIKM.

[57]  Kai-Wei Chang,et al.  Multi-Task Learning for Document Ranking and Query Suggestion , 2018, International Conference on Learning Representations.

[58]  Ben He,et al.  NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval , 2018, EMNLP.

[59]  Kevin Duh,et al.  Learning to rank with partially-labeled data , 2008, SIGIR '08.

[60]  W. Bruce Croft,et al.  From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing , 2018, CIKM.

[61]  Bhaskar Mitra,et al.  Cross Domain Regularization for Neural Ranking Models using Adversarial Learning , 2018, SIGIR.

[62]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[63]  Jian-Yun Nie,et al.  Multi-level Abstraction Convolutional Model with Weak Supervision for Information Retrieval , 2018, SIGIR.

[64]  Bhaskar Mitra,et al.  Report on the SIGIR 2016 Workshop on Neural Information Retrieval (Neu-IR) , 2016, SIGIR Forum.

[65]  W. Bruce Croft,et al.  aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model , 2016, CIKM.

[66]  W. Bruce Croft,et al.  Evaluating answer passages using summarization measures , 2014, SIGIR.

[67]  Ming-Wei Chang,et al.  A Knowledge-Grounded Neural Conversation Model , 2017, AAAI.

[68]  W. Bruce Croft,et al.  Relevance-based Word Embedding , 2017, SIGIR.

[69]  W. Bruce Croft,et al.  Learning a Deep Listwise Context Model for Ranking Refinement , 2018, SIGIR.

[70]  Idan Szpektor,et al.  Learning from the past: answering new questions with past answers , 2012, WWW.

[71]  Alessandro Moschitti,et al.  Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks , 2015, SIGIR.

[72]  Jimmy J. Lin,et al.  Simple Applications of BERT for Ad Hoc Document Retrieval , 2019, ArXiv.

[73]  Rui Yan,et al.  Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System , 2016, SIGIR.

[74]  Enhong Chen,et al.  Context-aware ranking in web search , 2010, SIGIR '10.

[75]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[76]  Pavel Serdyukov,et al.  Personalization of web-search using short-term browsing context , 2013, CIKM.

[77]  Hang Li,et al.  An Information Retrieval Approach to Short Text Conversation , 2014, ArXiv.

[78]  Xueqi Cheng,et al.  Text Matching as Image Recognition , 2016, AAAI.

[79]  Ben He,et al.  Training query filtering for semi-supervised learning to rank with pseudo labels , 2015, World Wide Web.

[80]  Ya Zhang,et al.  Multi-task learning for boosting with application to web search ranking , 2010, KDD.

[81]  Jun Xu,et al.  Modeling Diverse Relevance Patterns in Ad-hoc Retrieval , 2018, SIGIR.

[82]  Tie-Yan Liu,et al.  Listwise approach to learning to rank: theory and algorithm , 2008, ICML '08.

[83]  Stephen E. Robertson,et al.  SoftRank: optimizing non-smooth rank metrics , 2008, WSDM '08.

[84]  Zhiyuan Liu,et al.  Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search , 2018, WSDM.

[85]  Xuanjing Huang,et al.  Convolutional Neural Tensor Network Architecture for Community-Based Question Answering , 2015, IJCAI.

[86]  Hang Li,et al.  A Deep Architecture for Matching Short Texts , 2013, NIPS.

[87]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[88]  Yelong Shen,et al.  A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval , 2014, CIKM.

[89]  Siu Cheung Hui,et al.  Hyperbolic Representation Learning for Fast and Efficient Neural Question Answering , 2017, WSDM.

[90]  W. Bruce Croft,et al.  WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval , 2018, SIGIR.

[91]  Jun Huang,et al.  Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems , 2018, SIGIR.

[92]  Xiaodong Liu,et al.  Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval , 2015, NAACL.

[93]  Zhiguo Wang,et al.  Bilateral Multi-Perspective Matching for Natural Language Sentences , 2017, IJCAI.

[94]  Yi Yang,et al.  WikiQA: A Challenge Dataset for Open-Domain Question Answering , 2015, EMNLP.

[95]  Gerard de Melo,et al.  A Position-Aware Deep Model for Relevance Matching in Information Retrieval , 2017, ArXiv.

[96]  Larry P. Heck,et al.  Learning deep structured semantic models for web search using clickthrough data , 2013, CIKM.

[97]  Jianfeng Gao,et al.  Deep Learning for Natural Language Processing: Theory and Practice (Tutorial) , 2014 .

[98]  Jimmy J. Lin,et al.  Noise-Contrastive Estimation for Answer Selection with Deep Neural Networks , 2016, CIKM.

[99]  Jun Zhao,et al.  Inner Attention based Recurrent Neural Networks for Answer Selection , 2016, ACL.

[100]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[101]  Alan Ritter,et al.  Data-Driven Response Generation in Social Media , 2011, EMNLP.

[102]  Wei Liu,et al.  Neural Compatibility Modeling with Attentive Knowledge Distillation , 2018, SIGIR.

[103]  Shuohang Wang,et al.  A Compare-Aggregate Model for Matching Text Sequences , 2016, ICLR.

[104]  Xuehua Shen,et al.  Context-sensitive information retrieval using implicit feedback , 2005, SIGIR '05.

[105]  Wei Chu,et al.  Modeling the impact of short- and long-term behavior on search personalization , 2012, SIGIR '12.

[106]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[107]  Marc Najork,et al.  Learning Groupwise Scoring Functions Using Deep Neural Networks , 2018, ArXiv.

[108]  W. Bruce Croft,et al.  User Intent Prediction in Information-seeking Conversations , 2019, CHIIR.

[109]  Rui Yan,et al.  "Shall I Be Your Chat Companion?": Towards an Online Human-Computer Conversation System , 2016, CIKM.

[110]  Bhaskar Mitra,et al.  SIGIR 2017 Workshop on Neural Information Retrieval (Neu-IR'17) , 2017, SIGIR.

[111]  Hao Wang,et al.  A Dataset for Research on Short-Text Conversations , 2013, EMNLP.

[112]  Xueqi Cheng,et al.  RI-Match: Integrating Both Representations and Interactions for Deep Semantic Matching , 2018, AIRS.

[113]  Hang Li.  Learning to Rank for Information Retrieval and Natural Language Processing , 2011, Synthesis Lectures on Human Language Technologies.

[114]  Yiqun Liu,et al.  Relevance Estimation with Multiple Information Sources on Search Engine Result Pages , 2018, CIKM.

[115]  Jimmy J. Lin,et al.  Pseudo test collections for learning web search ranking functions , 2011, SIGIR.

[116]  J. Shane Culpepper,et al.  Neural Query Performance Prediction using Weak Supervision from Multiple Signals , 2018, SIGIR.

[117]  Christopher J. C. Burges,et al.  From RankNet to LambdaRank to LambdaMART: An Overview , 2010 .

[118]  Lei Yu,et al.  Deep Learning for Answer Sentence Selection , 2014, ArXiv.

[119]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[120]  Ji Wan,et al.  Deep Learning for Content-Based Image Retrieval: A Comprehensive Study , 2014, ACM Multimedia.

[121]  Geoffrey E. Hinton,et al.  Semantic hashing , 2009, Int. J. Approx. Reason..

[122]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[123]  Hamed Zamani,et al.  Situational Context for Ranking in Personal Search , 2017, WWW.

[124]  Tie-Yan Liu,et al.  Ranking Measures and Loss Functions in Learning to Rank , 2009, NIPS.

[125]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[126]  M. de Rijke,et al.  Neural Vector Spaces for Unsupervised Information Retrieval , 2017, ACM Trans. Inf. Syst..

[127]  Jimmy J. Lin,et al.  Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search , 2018, AAAI.

[128]  Md. Mustafizur Rahman,et al.  Neural information retrieval: at the end of the early years , 2017, Information Retrieval Journal.

[129]  Rabab Kreidieh Ward,et al.  Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[130]  Tie-Yan Liu,et al.  Word-Entity Duet Representations for Document Ranking , 2017, SIGIR.

[131]  Tetsuya Sakai,et al.  Overview of the NTCIR-12 Short Text Conversation Task , 2016, NTCIR.

[132]  Dong-Hong Ji,et al.  Multi-Granularity Neural Sentence Model for Measuring Short Text Similarity , 2017, DASFAA.

[133]  Noah A. Smith,et al.  What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA , 2007, EMNLP.

[134]  Bowen Zhou,et al.  Applying deep learning to answer selection: A study and an open task , 2015, 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).

[135]  Grace Hui Yang,et al.  DeepTileBars: Visualizing Term Distribution for Neural Information Retrieval , 2019, AAAI.

[136]  Siu Cheung Hui,et al.  Learning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture , 2017, SIGIR.

[137]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[138]  Diego Molla Aliod,et al.  Question Answering in Restricted Domains: An Overview , 2007, CL.

[139]  Timothy Baldwin,et al.  CQADupStack: A Benchmark Data Set for Community Question-Answering Research , 2015, ADCS.

[140]  Le Zhao,et al.  Term necessity prediction , 2010, CIKM.

[141]  W. Bruce Croft,et al.  Embedding-based Query Language Models , 2016, ICTIR.

[142]  Zhiyuan Liu,et al.  Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval , 2018, ACL.

[143]  Zhoujun Li,et al.  Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots , 2016, ArXiv.

[144]  Zhen Xu,et al.  Incorporating loose-structured knowledge into conversation modeling via recall-gate LSTM , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[145]  Xueqi Cheng,et al.  A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations , 2015, AAAI.

[146]  Yang Deng,et al.  Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs , 2018, SIGIR.

[147]  Gerard de Melo,et al.  PACRR: A Position-Aware Neural IR Model for Relevance Matching , 2017, EMNLP.

[148]  Xueqi Cheng,et al.  Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN , 2016, IJCAI.

[149]  Bhaskar Mitra,et al.  Neural Models for Information Retrieval , 2017, ArXiv.

[150]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[151]  Fabrizio Silvestri,et al.  Context- and Content-aware Embeddings for Query Rewriting in Sponsored Search , 2015, SIGIR.

[152]  Filip Radlinski,et al.  Personalizing web search using long term browsing history , 2011, WSDM '11.

[153]  Dongyan Zhao,et al.  Joint Learning of Response Ranking and Next Utterance Suggestion in Human-Computer Conversation System , 2017, SIGIR.

[154]  Brendan T. O'Connor,et al.  Understanding the Representational Power of Neural Retrieval Models Using NLP Tasks , 2018, ICTIR.

[155]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[156]  Fernando Diaz,et al.  SIGIR 2018 Workshop on Learning from Limited or Noisy Data for Information Retrieval , 2018, SIGIR.

[157]  Azadeh Shakery,et al.  Pseudo-Relevance Feedback Based on Matrix Factorization , 2016, CIKM.

[158]  James Allan,et al.  Universal Approximation Functions for Fast Learning to Rank: Replacing Expensive Regression Forests with Simple Feed-Forward Networks , 2018, SIGIR.

[159]  Di Wang,et al.  A Long Short-Term Memory Model for Answer Sentence Selection in Question Answering , 2015, ACL.

[160]  W. Bruce Croft,et al.  Learning a Hierarchical Embedding Model for Personalized Product Search , 2017, SIGIR.

[161]  Laure Soulier,et al.  Toward a Deep Neural Approach for Knowledge-Based IR , 2016, SIGIR 2016.

[162]  Stephen E. Robertson,et al.  Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval , 1994, SIGIR '94.