Performance evaluation of search result diversification methods and their stability

Recently search result diversification has attracted a lot of attention in the information retrieval and Web search research community. Although quite a few result diversification algorithms have been proposed, it lacks in comparative study on those algorithms. In this paper, we compare four explicit result diversification algorithms. Three of them are representative explicit result diversification methods xQuAD, PM2 and IA-SELECT. The fourth, CombSumDiv, is presented in this paper. The experiments are carried out with 4 data sets used in the web track of TREC. The experimental results show that the proposed method CombSumDiv is as good as xQuAD, while PM2 and IA-SELECT are not as good as the other two. Stability of the four algorithms are also tested in a variety of situations. Experiments show that all four result diversification methods have very good stability when the search condition is not ideal.

[1]  M. de Rijke,et al.  Result diversification based on query-specific cluster ranking , 2011, J. Assoc. Inf. Sci. Technol..

[2]  Craig MacDonald,et al.  Intent-aware search result diversification , 2011, SIGIR.

[3]  Hui Fang,et al.  A Comparative Study of Search Result Diversification Methods , 2011 .

[4]  Craig MacDonald,et al.  Explicit Search Result Diversification through Sub-queries , 2010, ECIR.

[5]  Emre Velipasaoglu,et al.  Intent-based diversification of web search results: metrics and algorithms , 2011, Information Retrieval.

[6]  W. Bruce Croft,et al.  Diversity by proportionality: an election-based approach to search result diversification , 2012, SIGIR '12.

[7]  Ben Carterette,et al.  An analysis of NP-completeness in novelty and diversity ranking , 2009, Information Retrieval.

[8]  Sreenivas Gollapudi,et al.  Diversifying search results , 2009, WSDM '09.

[9]  W. Bruce Croft,et al.  Term level search result diversification , 2013, SIGIR.

[10]  Wei Zheng,et al.  A Diagnostic Study of Search Result Diversification Methods , 2013, ICTIR.

[11]  Jun Wang,et al.  Portfolio theory of information retrieval , 2009, SIGIR.

[12]  Xueqi Cheng,et al.  Learning for search result diversification , 2014, SIGIR.

[13]  Jade Goldstein-Stewart,et al.  The use of MMR, diversity-based reranking for reordering documents and producing summaries , 1998, SIGIR '98.

[14]  Jade Goldstein-Stewart,et al.  The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries , 1998, SIGIR Forum.

[15]  Maarten de Rijke,et al.  A Query Performance Analysis for Result Diversification , 2011, ICTIR.

[16]  Charles L. A. Clarke,et al.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods , 2009, SIGIR.

[17]  Shengli Wu,et al.  Search result diversification via data fusion , 2014, SIGIR.

[18]  Craig MacDonald,et al.  Exploiting query reformulations for web search result diversification , 2010, WWW '10.

[19]  Hong Cheng,et al.  Coverage-based search result diversification , 2012, Information Retrieval.

[20]  Saul Vargas,et al.  Explicit relevance models in intent-oriented information retrieval diversification , 2012, SIGIR '12.

[21]  Charles L. A. Clarke,et al.  The impact of intent selection on diversified search evaluation , 2013, SIGIR.

[22]  Mark Sanderson,et al.  Using score differences for search result diversification , 2014, SIGIR.

[23]  Wenfei Fan,et al.  On the Complexity of Query Result Diversification , 2013, Proc. VLDB Endow..

[24]  Fuji Ren,et al.  Search Result Diversification via Filling Up Multiple Knapsacks , 2014, CIKM.