Understanding Ephemeral State of Relevance

Despite its dynamic nature, relevance is often measured in a context-independent manner in information retrieval practice. We look into this discrepancy. We propose a contextual relevance/usefulness measurement called ephemeral state of relevance (ESR), which is defined as the amount of useful information a user acquired from a clicked result as assessed just after examining the result during an interactive search session. We collect ESR and context-independent usefulness judgments through a laboratory user study and compare the two. We examine factors related to both judgments and examine their differences. Our study demonstrates a few advantages of ESR: it captures users' real-time state of mind and perceptions; it measures how much useful information the user is able to acquire from a result rather than how much there is in the result; it better reflects users' needs and criteria of useful results during a session, highlighting novelty as a salient factor. However, we also find that users may not be able to correctly assess the credibility of information during a session, which may reduce the reliability of the collected ESR judgments. We evaluate ESR, context-independent usefulness judgments, and TREC-style topical relevance judgments by correlating with user experience in a session. The results demonstrate that switching the judgment criterion from topical relevance to usefulness is fruitful, but moving from context-independent judgments to contextual ones has only limited advantages with respect to its cost and complexity. Our study enriches current understanding on the dynamics of relevance in a search session and identifies both opportunities and challenges for collecting contextual relevance judgments.