GEOGRAPHIC IR SYSTEMS: REQUIREMENTS AND EVALUATION

Geographic information retrieval is a new and evolving domain. The development of GIR systems requires the analysis of requirements for such systems and, after systems are implemented their evaluation. This paper describes requirement analysis and evaluation for the SPIRIT system. Methods of user and system-centred evaluation are described and a methodology for building a document collection to facilitate derivation of measures of system performance, together with a new scheme for the evaluation of spatial and thematic relevance with respect to search results are introduced. The paper stresses the importance of developing approaches to evaluate GIR systems which holistically evaluate user interactions as well as measures of system performance. Introduction Geographic information retrieval (GIR) is a fast developing area “concerned with providing access to geo-referenced information sources” (Larson, 1996). In recent years information retrieval has become to a large extent synonymous with the retrieval of relevant documents from large collections of unstructured text-based documents stored on the web. In this context, we will define geographic information retrieval more narrowly than Larson, as the retrieval of geographically and thematically relevant documents in response to a query of the form (e.g. Castles, Scotland), where the spatial relationship may either implicitly imply containment, or explicitly be selected from a set of possible topological, proximity and directional options (e.g. inside, near, north of) and where the documents searched are those available on the web. Developments in GIR are driven both by academic enquiry and commercial interests, with for example Google having recently introduced a so-called “Local” search engine based on the integration of data provided by business directories and web documents (http://local.google.co.uk/).