From Highly Relevant to Not Relevant: Examining Different Regions of Relevance

Abstract User relevance judgments are central to both the systems and user-oriented approaches to information retrieval (IR) systems research and development. User-oriented relevance research has also operated on two largely unconnected tracks. First, a relevance level track that examines users' criteria for relevance judgments. Second, a regions of relevance track that examines the measurement of users' relevance judgments. Users judgments and criteria for highly relevant items have been central issues for much of the relevance research. Findings are presented from four separate studies of relevance judgments by 55 users, conducting their initial online search on a particular information problem. In three studies, the number of items judged “partially” relevant (on a scale of relevant, partially relevant or not relevant) was positively correlated with different aspects of changes in users', including: (1) information problem definition, (2) search intermediaries' perceptions that a user's question and information problem has changed during the mediated search interaction, (3) personal knowledge due to the search interaction, and (4) criteria for making relevance judgments. Users with high knowledge and topic levels were more likely to judge items as highly relevant. Differences between users' criteria for highly, partially and non-relevant items are also identified. Findings suggest the need to expand the framework for relevance research and further identify the characteristics of the middle region of relevance or partial relevance as: (1) partially relevant items may play an important role in the early stages of a user's information seeking process over time for a particular information problem and (2) a relationship may exist between partially relevant items retrieved and changes in users' information problems during an information seeking process. Results also suggest that partially relevant items may be useful at the early stages of users' information seeking processes. We propose a useful concept of relevance as a relationship and an effect on the movement of a user through the iterative stages of their information seeking process. Users' relevance judgments can also be plotted on a three-dimensional spatial model of relevance level, region and time. Implications for the development of IR systems, searching practice and relevance research are also discussed.