User Expressions of Relevance Judgment Certainty

It is well known that different people produce different judgments regarding the relevance of documents to given search topics. What is less known are the reasons for such differences. We report preliminary results from a user study showing that users can have uncertainty in their judgments. At the extreme, such uncertainty results in the user making a guess. Fortunately, users are aware of their uncertainty. Relevance judging interfaces should be designed to capture not only a user’s relevance judgment but also the user’s certainty in their judgment.