An Initial Proposal for Cooperative Evaluation on Information Retrieval in Portuguese

In this paper we discuss evaluation of information retrieval, Web search and question answering systems, paving the way for the organization of an evaluation contest on IR for Portuguese. Inspired by current international setups, we motivate the need to study the specific problems posed by Portuguese, suggesting a collection suitable for multiple tasks.

[1]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[2]  Oren Etzioni,et al.  Web document clustering: a feasibility demonstration , 1998, SIGIR '98.

[3]  Tim Bray,et al.  Measuring the Web , 1996, World Wide Web J..

[4]  Peter Bailey,et al.  Measuring Search Engine Quality , 2001, Information Retrieval.

[5]  Noriko Kando,et al.  Overview of the Web Retrieval Task at the Third NTCIR Workshop , 2003, NTCIR.

[6]  Jonathan Ginzburg,et al.  Proceedings of COLING 2004 , 2004 .

[7]  Joseph B. Walther,et al.  The Value of Web Log Data in Use-Based Design and Testing , 2006, J. Comput. Mediat. Commun..

[8]  Mário J. Silva,et al.  The Case for a Portuguese Web Search Engine , 2003, ICWI.

[9]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[10]  Thomas Mandl,et al.  Linguistic and Statistical Analysis of the CLEF Topics , 2002, CLEF.

[11]  E. Garfield.  Citation analysis as a tool in journal evaluation. , 1972, Science.

[12]  Ralph Grishman,et al.  Message Understanding Conference- 6: A Brief History , 1996, COLING.

[13]  Henry G. Small,et al.  Co-citation in the scientific literature: A new measure of the relationship between two documents , 1973, J. Am. Soc. Inf. Sci..

[14]  Diana Santos,et al.  Measuring the Web in Portuguese , 2002 .

[15]  Christian R. Huyck,et al.  Portuguese-English Experiments using Latent Semantic Indexing , 2002, CLEF.

[16]  Lynette Hirschman,et al.  The Evolution of evaluation: Lessons from the Message Understanding Conferences , 1998, Comput. Speech Lang..

[17]  Ellen M. Voorhees,et al.  Overview of TREC 2001 , 2001, TREC.

[18]  Sriram Raghavan,et al.  Searching the Web , 2001, ACM Trans. Internet Techn..

[19]  Monika Henzinger,et al.  Finding Related Pages in the World Wide Web , 1999, Comput. Networks.

[20]  Paulo Rocha,et al.  AvalON: uma iniciativa de avaliação conjunta para o português , 2003 .

[21]  Paulo Quaresma,et al.  PGR: Portuguese Attorney General's Office Decisions on the Web , 2001, INAP.

[22]  Thorsten Joachims,et al.  Evaluating Retrieval Performance Using Clickthrough Data , 2003, Text Mining.

[23]  Jacek Gwizdka,et al.  Towards Information Retrieval Measures for Evaluation of Web Search Engines , 1999 .