Overview of the Informational Retrieval Task at NTCIR-4 WEB

This paper gives an overview of the Informational Retrieval Task 2 that was conducted from 2003 to 2004 as a subtask of the WEB Task at the Fourth NTCIR Workshop (‘NTCIR-4 WEB’). In the Informational Retrieval Task, we attempted to assess the retrieval effectiveness of Web search engine systems from a viewpoint of topical relevance, and to build a re-usable test collection suitable for evaluating Web search engine systems from such a viewpoint. We used 100gigabyte document data that were mainly gathered from the ‘.jp’ domain. Relevance judgments were performed on the retrieved documents, which were written in Japanese or English, by considering the relationshiop between the pages referenced by hyper-links. We also investigated the evaluation methods considering non-redundancy of contents and diversity of queries.