Methods for comparing rankings of search engine results

In this paper we present a number of measures that compare rankings of search engine results. We apply these measures to five queries that were monitored daily for two periods of 14 or 21 days each. Rankings of the different search engines (Google, Yahoo! and Teoma for text searches and Google, Yahoo! and Picsearch for image searches) are compared on a daily basis, in addition to longitudinal comparisons of the same engine for the same query over time. The results and rankings of the two periods are compared as well.

[1]  Peter Bailey,et al.  Measuring Search Engine Quality , 2001, Information Retrieval.

[2]  Ian Soboroff,et al.  Ranking retrieval systems without relevance judgments , 2001, SIGIR '01.

[3]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[4]  Thorsten Joachims,et al.  Evaluating Retrieval Performance Using Clickthrough Data , 2003, Text Mining.

[5]  Judit Bar-Ilan,et al.  Dynamics of Search Engine Rankings - A Case Study , 2004, WebDyn@WWW.

[6]  Andrei Z. Broder,et al.  A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines , 1998, Comput. Networks.

[7]  HenzingerMonika,et al.  Analysis of a very large web search engine query log , 1999 .

[8]  Ronald Fagin,et al.  Comparing top k lists , 2003, SODA '03.

[9]  Amanda Spink,et al.  A temporal comparison of AltaVista Web searching , 2005, J. Assoc. Inf. Sci. Technol..

[10]  M. M. Sufyan Beg.  A subjective measure of web search quality , 2005, Inf. Sci..

[11]  C. Lee Giles,et al.  Accessibility of information on the web , 1999, Nature.

[12]  Lisa Zhao.  Jump Higher: Analyzing Web-Site Rank in Google , 2004 .

[13]  Amanda Spink,et al.  Model for organizational knowledge creation and strategic use of information: Research Articles , 2005 .

[14]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[15]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[16]  Bernard J. Jansen,et al.  Coverage, relevance, and ranking: The impact of query operators on Web search engine results , 2003, TOIS.

[17]  Stefano Mizzaro,et al.  Relevance: The Whole History , 1997, J. Am. Soc. Inf. Sci..

[18]  Liwen Vaughan,et al.  New measurements for search engine evaluation proposed and tested , 2004, Inf. Process. Manag..

[19]  Antonio Gulli,et al.  The indexable web is more than 11.5 billion pages , 2005, WWW '05.

[20]  Amanda Spink,et al.  U.S. versus European web searching trends , 2002, SIGF.

[21]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[22]  Judit Bar-Ilan,et al.  Comparing rankings of search results on the Web , 2005, Inf. Process. Manag..

[23]  M. Lemay.  coverage , 1972, cell tower.

[24]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[25]  Amanda Spink,et al.  An analysis of Web searching by European AlltheWeb.com users , 2005, Inf. Process. Manag..

[26]  Louise T. Su.  A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates , 2003, J. Assoc. Inf. Sci. Technol..

[27]  Michael D. Gordon,et al.  Finding Information on the World Wide Web: The Retrieval Effectiveness of Search Engines , 1999, Inf. Process. Manag..

[28]  R. Graham,et al.  Spearman's Footrule as a Measure of Disarray , 1977 .

[29]  Ingmar Weber,et al.  An Analysis of Factors Used in Search Engine Ranking , 2005, AIRWeb.

[30]  Hsin-Liang Chen,et al.  Evaluation of Web-Based Search Engines from the End-User's Perspective: A Pilot Study , 1998 .

[31]  Abdur Chowdhury,et al.  Automatic evaluation of world wide web search services , 2002, SIGIR '02.