First 20 precision among World Wide Web search services (search engines)

Five search engines, Alta Vista, Excite, HotBot, Infoseek, and Lycos, are compared for precision on the first 20 results returned for 15 queries, adding weight for ranking effectiveness. All searching was done from January 31 to March 12, 1997. In the study, steps are taken to ensure that bias has not unduly influenced the evaluation. Friedmann’s randomized block design is used to perform multiple comparisons for significance. Analysis shows that Alta Vista, Excite and Infoseek are the top three services, with their relative rank changing depending on how one operationally defines the concept of relevance. Correspondence analysis shows that Lycos performed better on short, unstructured queries, whereas HotBot performed better on structured queries.

[1]  Donna K. Harman,et al.  Overview of the Fourth Text REtrieval Conference (TREC-4) , 1995, TREC.

[2]  Louise T. Su.  The Relevance of Recall and Precision in User Evaluation , 1994, J. Am. Soc. Inf. Sci..

[3]  Terry Noreault,et al.  Automatic ranked output from boolean searches in SIRE , 1977, J. Am. Soc. Inf. Sci..

[4]  Carolyn Snyder,et al.  Web Site Usability: A Designer's Guide , 1997 .

[5]  Guijun Wang,et al.  Information fusion with ProFusion , 1996, WebNet.

[6]  Donna K. Harman,et al.  Overview of the Second Text REtrieval Conference (TREC-2) , 1994, HLT.

[7]  Gary Marchionini,et al.  A Comparative Study of Web Search Service Performance , 1996 .

[8]  Nicholas G. Tomaiuolo,et al.  An analysis of Internet search engines: assessment of over 200 search queries , 1996 .

[9]  Donna Harman,et al.  The fourth text REtrieval conference , 1996 .

[10]  Hao-hua Chu,et al.  Search En-gines for the World Wide Web: A Compara-tive Study and Evaluation Methodology , 1996 .

[11]  Alfred Glossbrenner,et al.  Search engines for the World Wide Web , 1997 .

[12]  Giles,et al.  Searching the world wide Web , 1998, Science.

[13]  R. A. Groeneveld,et al.  Practical Nonparametric Statistics (2nd ed). , 1981 .

[14]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[15]  Stefano Mizzaro,et al.  Relevance: The Whole History , 1997, J. Am. Soc. Inf. Sci..

[16]  T. Obremski.  Practical Nonparametric Statistics (2nd ed.) , 1981 .

[17]  Peter Willett,et al.  Estimating the recall performance of Web search engines , 1997 .

[18]  David Moxley,et al.  Authoritative Guide to Web Search Engines , 1997 .