The evaluation of WWW search engines

The literature of the evaluation of Internet search engines is reviewed. Although there have been many studies, there has been little consistency in the way such studies have been carried out. This problem is exacerbated by the fact that recall is virtually impossible to calculate in the fast changing Internet environment, and therefore the traditional Cranfield type of evaluation is not usually possible. A variety of alternative evaluation methods has been suggested to overcome this difficulty. The authors recommend that a standardised set of tools is developed for the evaluation of web search engines so that, in future, comparisons can be made between search engines more effectively, and that variations in performance of any given search engine over time can be tracked. The paper itself does not provide such a standard set of tools, but it investigates the issues and makes preliminary recommendations of the types of tools needed.

[1]  Vijay V. Raghavan,et al.  Information Retrieval on the World Wide Web , 1997, IEEE Internet Comput..

[2]  Peter Willett,et al.  Estimating the recall performance of Web search engines , 1997 .

[3]  Frank Teuteberg.  Effektives Suchen im World Wide Web: Suchdienste und Suchmethoden , 1997, Wirtsch..

[4]  Ian Winship.  World‐Wide Web searching tools: an evaluation , 1995 .

[5]  Stephen P. Harter,et al.  Evaluation of information retrieval systems : Approaches, issues, and methods , 1997 .

[6]  Diane Nahl,et al.  Ethnography of novices' first use of Web search engines: affective control in cognitive processing , 1998 .

[7]  Peter Burden,et al.  Searching the world wide web: an evaluation of available tools and methodologies , 1999, Inf. Softw. Technol..

[8]  Guijun Wang,et al.  Information fusion with ProFusion , 1996, WebNet.

[9]  Martin Frické,et al.  Measuring recall , 1998, J. Inf. Sci..

[10]  Mark Kerr.  Universal access, by design , 1997 .

[11]  Jaideep Srivastava,et al.  First 20 precision among World Wide Web search services (search engines) , 1999 .

[12]  Candy Schwartz,et al.  Web Search Engines , 1998, J. Am. Soc. Inf. Sci..

[13]  Andrei Z. Broder,et al.  A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines , 1998, Comput. Networks.

[14]  安形 輝,et al.  インターネットのサーチエンジンの評価尺度 : ESL (Expected Search Length) を使った検索実験 , 1997 .

[15]  Charles Oppenheim,et al.  EVALUATIONS OF ELECTRONIC DATABASES, PART I: CRITERIA FOR TESTING CDROM PRODUCTS , 1993 .

[16]  Xiaoying Dong,et al.  SEARCH ENGINES ON THE WORLD WIDE WEB AND INFORMATION RETRIEVAL FROM THE INTERNET: A REVIEW AND EVALUATION , 1997 .

[17]  David Ellis,et al.  Progress and problems in information retrieval , 1996 .

[18]  Heather Tunender,et al.  How To Succeed in Promoting Your Web Site: The Impact of Search Engine Registration on Retrieval of a World Wide Web Site. , 1998 .

[19]  Nicholas G. Tomaiuolo,et al.  An analysis of Internet search engines: assessment of over 200 search queries , 1996 .

[20]  Stuart I. Feldman.  Web search services in 1998: Trends and challenges , 1998 .

[21]  C. Lee Giles,et al.  Searching the Web: general and scientific information access , 1999, First IEEE/POPOV Workshop on Internet Technologies and Services. Proceedings (Cat. No.99EX391).

[22]  Charles Oppenheim,et al.  EVALUATION OF ELECTRONIC DATABASES, PART II: TESTING CDROM PRODUCTS , 1993 .

[23]  W. S. Cooper.  Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems , 1968 .

[24]  Tim Bray,et al.  Measuring the Web , 1996, World Wide Web J..

[25]  Chris Armstrong,et al.  Metadata, recall, and abstracts: can abstracts ever be reliable indicators of document value? , 1997 .

[26]  Hao-hua Chu,et al.  Search En-gines for the World Wide Web: A Compara-tive Study and Evaluation Methodology , 1996 .

[27]  Pasquale Savino,et al.  Multimedia Document Search on the Web , 1998, Comput. Networks.

[28]  Charles Oppenheim,et al.  Access to information on the World Wide Web for blind and visually impaired people , 1999 .

[29]  Stacey Kimmel.  Robot-Generated Databases on the World Wide Web. , 1996 .

[30]  J. Swets.  Signal detection and recognition by human observers : contemporary readings , 1964 .

[31]  Oleg Konstantinovich Tikhomirov,et al.  The psychology of thinking , 1988 .

[32]  Mark Chignell,et al.  From information retrieval to hypertext and back again: the role of interaction in the information exploration interface , 1997 .

[33]  Christine A. DeZelar-Tiedman,et al.  Known-item searching on the World Wide Web , 1997 .

[34]  Jonathan Furner.  IR on the Web : an overview , 1996 .

[35]  Gary Marchionini,et al.  A Comparative Study of Web Search Service Performance , 1996 .

[36]  Giles,et al.  Searching the world wide Web , 1998, Science.

[37]  Amelia Kassel.  Media directories online , 1998 .

[38]  Jesper Vissing Laursen.  Somebody wants to get in touch with you: search engine persuasion , 1998 .

[39]  G. G. Chowdhury,et al.  The internet and information retrieval research: a brief review , 1999, J. Documentation.