Methods for measuring search engine performance over time

This study introduces methods for evaluating search engine performance over a time period. Several measures are defined, which as a whole describe search engine functionality over time. The necessary setup for such studies is described, and the use of these measures is illustrated through a specific example. The set of measures introduced here may serve as a guideline for the search engines for testing and improving their functionality. We recommend setting up a standard suite of measures for evaluating search engine performance.

[1]  Carol Ebbinghouse,et al.  Virtuous Funding for the Virtual Library: The Annual SCOUG Retreat, 1997. , 1997 .

[2]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[3]  Giles,et al.  Searching the world wide Web , 1998, Science.

[4]  Judit Bar-Ilan.  Evaluating the stability of the search tools Hotbot and Snap: a case study , 2000, Online Inf. Rev..

[5]  C. Lee Giles,et al.  Accessibility of information on the web , 1999, Nature.

[6]  Andrei Z. Broder,et al.  Graph structure in the Web , 2000, Comput. Networks.

[7]  Amanda Spink,et al.  Regions and levels: Measuring and mapping users' relevance judgments , 2001, J. Assoc. Inf. Sci. Technol..

[8]  Peter Willett,et al.  Estimating the recall performance of Web search engines , 1997 .

[9]  George Cybenko,et al.  How dynamic is the Web? , 2000, Comput. Networks.

[10]  Andrei Z. Broder,et al.  A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines , 1998, Comput. Networks.

[11]  Wallace Koehler,et al.  An Analysis of Web Page and Web Site Constancy and Permanence , 1999, J. Am. Soc. Inf. Sci..

[12]  Gerald Salton,et al.  Automatic text processing , 1988 .

[13]  Gary Stixon,et al.  Japan Fields a Big-League Light Gatherer , 1999 .

[14]  Donna K. Harman,et al.  Overview of the Ninth Text REtrieval Conference (TREC-9) , 2000, Text Retrieval Conference.

[15]  Edward T. O'Neill,et al.  A Methodology for Sampling the World Wide Web , 2001 .

[16]  Jaideep Srivastava,et al.  First 20 precision among World Wide Web search services (search engines) , 1999 .

[17]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[18]  Michael D. Gordon,et al.  Finding Information on the World Wide Web: The Retrieval Effectiveness of Search Engines , 1999, Inf. Process. Manag..

[19]  C. J. van Rijsbergen,et al.  FOUNDATION OF EVALUATION , 1974 .