Evaluating the effectiveness of Web search engines on results diversification

Introduction. Recently, the problem of diversification of search results has attracted a lot of attention in the information retrieval and Web search research community. For multi-faceted or ambiguous queries, a search engine is generally favoured if it is able to identify relevant documents on a wider range of different aspects. Method. We evaluate the performance of three major Web search engines: Google, Bing and Ask manually using 200 multi-faceted or ambiguous queries from TREC. Analysis. Both classical metrics and intent-aware metrics are used to evaluate search results. Results. Experimental results show that on average Bing and Google are comparable and Ask is slightly worse than the former two. However, Ask does very well in one subtype of queries – ambiguous queries. The average performance of the three search engines is better than the average of the top two runs submitted to the TREC web diversity task in 2009-2012. Conclusions. Generally, all three Web search engines do well, this indicates that all of them must use state-of-the-art technology to support the diversification of search results.

[1]  Jaideep Srivastava,et al.  First 20 precision among World Wide Web search services (search engines) , 1999 .

[2]  Olivier Chapelle,et al.  Expected reciprocal rank for graded relevance , 2009, CIKM.

[3]  Alfred Glossbrenner,et al.  Search engines for the World Wide Web , 1997 .

[4]  Jade Goldstein-Stewart,et al.  The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries , 1998, SIGIR Forum.

[5]  Tetsuya Sakai,et al.  Evaluating Search Result Diversity using Intent Hierarchies , 2016, SIGIR.

[6]  Rabia Nuray-Turan,et al.  Automatic performance evaluation of Web search engines , 2004, Inf. Process. Manag..

[7]  Rakesh Chandra Balabantaray.  Evaluation of web search engine based on ranking of results and its features , 2017 .

[8]  Pertti Vakkari,et al.  Comparing Google to a digital reference service for answering factual and topical requests by keyword and question queries , 2011, Online Inf. Rev..

[9]  M. de Rijke,et al.  Result diversification based on query-specific cluster ranking , 2011, J. Assoc. Inf. Sci. Technol..

[10]  Michael D. Gordon,et al.  Finding Information on the World Wide Web: The Retrieval Effectiveness of Search Engines , 1999, Inf. Process. Manag..

[11]  Xueqi Cheng,et al.  Directly Optimize Diversity Evaluation Measures , 2017, ACM Trans. Intell. Syst. Technol..

[12]  Gary Marchionini,et al.  A Comparative Study of Web Search Service Performance , 1996 .

[13]  Thomas Mandl,et al.  Evaluation of five web search engines in Arabic language , 2010, LWA.

[14]  Shengli Wu,et al.  Effectiveness Evaluation and Comparison of Web Search Engines and Meta-search Engines , 2004, WAIM.

[15]  Rajesh Kumar Goutam,et al.  Performance Evaluation of search engines via user efforts P erformance Evaluation of search engines via user efforts Performance Evaluation of search engines via user efforts Performance Evaluation of search engines via user efforts measures , 2012 .

[16]  Wei Zheng,et al.  A Diagnostic Study of Search Result Diversification Methods , 2013, ICTIR.

[17]  Jun Wang,et al.  Portfolio theory of information retrieval , 2009, SIGIR.

[18]  W. Bruce Croft,et al.  Diversity by proportionality: an election-based approach to search result diversification , 2012, SIGIR '12.

[19]  Roi Blanco,et al.  An in-depth study on diversity evaluation: The importance of intrinsic diversity , 2017, Inf. Process. Manag..

[20]  Dirk Lewandowski,et al.  Evaluating the retrieval effectiveness of web search engines using a representative query sample , 2014, J. Assoc. Inf. Sci. Technol..

[21]  John D. Lafferty,et al.  Beyond independent relevance: methods and evaluation metrics for subtopic retrieval , 2003, SIGIR.

[22]  Soon Ae Chun,et al.  A prediction model for web search hit counts using word frequencies , 2011, J. Inf. Sci..

[23]  B. T. Sampath Kumar,et al.  Evaluating the searching capabilities of search engines and metasearch engines: a comparative study , 2010 .

[24]  Tetsuya Sakai,et al.  Search Result Diversification Based on Hierarchical Intents , 2015, CIKM.

[25]  Amanda Spink,et al.  Real life information retrieval: a study of user queries on the Web , 1998, SIGF.

[26]  Farooq Ahmad,et al.  A survey on search results diversification techniques , 2015, Neural Computing and Applications.

[27]  Krishna Bharat,et al.  Diversifying web search results , 2010, WWW '10.

[28]  M. de Rijke,et al.  Fusion helps diversification , 2014, SIGIR.

[29]  Ji-Rong Wen,et al.  Learning to Diversify Search Results via Subtopic Attention , 2017, SIGIR.

[30]  Craig MacDonald,et al.  Exploiting query reformulations for web search result diversification , 2010, WWW '10.

[31]  Donna K. Harman,et al.  Results and Challenges in Web Search Evaluation , 1999, Comput. Networks.

[32]  Ahmet Uyar,et al.  Investigation of the accuracy of search engine hit counts , 2009, J. Inf. Sci..

[33]  Ahmet Uyar,et al.  Investigating the precision of Web image search engines for popular and less popular entities , 2017, J. Inf. Sci..

[34]  Adrian Iftene,et al.  Diversifying Search Results Using Semantic Resources , 2015, RoCHI.

[35]  Ben Carterette,et al.  Preference based evaluation measures for novelty and diversity , 2013, SIGIR.

[36]  Bernard J. Jansen,et al.  Coverage, relevance, and ranking: The impact of query operators on Web search engine results , 2003, TOIS.

[37]  Ying Liu,et al.  Evaluate and Compare Chinese Internet Search Engines Based on Users' Experience , 2007, 2007 International Conference on Wireless Communications, Networking and Mobile Computing.

[38]  Ben Carterette,et al.  Probabilistic models of ranking novel documents for faceted topic retrieval , 2009, CIKM.

[39]  Greg R. Notess,et al.  Searching the World-Wide Web: Lycos, WebCrawler and more , 1995 .

[40]  Sanjib Kumar Deka,et al.  Performance evaluation and comparison of the five most used search engines in retrieving web resources , 2010, Online Inf. Rev..