Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior

Abstract Research on relevance has established a conceptual consensus that stresses the importance of studying relevance judgments from a perspective that takes the users of retrieval systems into account. Yet little research has investigated how actual system users make relevance judgments. Theoretical claims pertaining to the nature of the relevance judgment process, thus, remain untested. This study is a step in the empirical exploration of the evolutionary nature of relevance judgments. The study intensively focuses on a single person with a real information problem. She was observed during both her online searching and document retrieval. The subject made her relevance evaluations first using bibliographic records and then using full-text documents as is typical of search processes that rely on bibliographic retrieval tools. The data consist of printouts of records and full-texts containing the subject's evaluation markings as well as transcripts of think-aloud protocols and her responses to questions during post-search interview sessions. The mental model concept is employed for analysis purposes and is operationalized as the research subject's changing perception of the information that she needs for her purposes as expressed in her relevance judgments. Specifically, the think-aloud protocols and markings of texts provide indications of the state of the subject's mental model and its change as she interacted with the materials that she retrieved and selected as relevant or possibly relevant. Frequencies of terms marked at the stage of record evaluation and the topical categories highlighted at the stage of document review were computed to provide a more concrete indication of the topical change in the subject's mental model of the needed information. The study also identified the set of judgment criteria that the subject applied during her evaluation of online records. Special attention during the analysis was paid to anomalous judgment behaviors demonstrated by the subject such as her deselection process in record evaluation and topic reformulation in document evaluation. Overall, the findings help untangle the relevance judgment process for one individual and one situation. In doing so, the findings provide a preliminary anchor for understanding the nature of the relevance judgment process of people engaged in an information search process. The subject's deselection of items and associated application of judgment criteria provide specific insights into how relevance judgment occurs.