Fuzzy versus probabilistic models for user relevance judgments

A crucial aspect of information retrieval is the process of making relevance judgments. Although this highly complex decision making procedure still eludes researchs, it is evident that a number of mental models are involved: models of the information need, retrieval system, database, user's knowledge in the subject area etc. In general, it is accepted that relevance judgments are made by evaluating documents for an overall conceptual match with the information need. In this study, we take the view that, given a document and a query, users first judge the document against the individual concepts in the query and then use some inferencing process to derive from these «atomic» decisions a «compound» judgment for the entire query. The probabilistic and fuzzy models are used as two points of reference against which to analyze the user's decision process in making compound judgments. The conclusions are interesting when comparing performances across AND and OR queries