Variations in relevance judgments and the measurement of retrieval effectiveness

Test collections have traditionally been used by information retrieval researchers to improve their retrieval strategies. To be viable as a laboratory tool, a collection must reliably rank diAerent retrieval variants according to their true eAectiveness. In particular, the relative eAectiveness of two retrieval strategies should be insensitive to modest changes in the relevant document set since individual relevance assessments are known to vary widely. The test collections developed in the TREC workshops have become the collections of choice in the retrieval research community. To verify their reliability, NIST investigated the eAect changes in the relevance assessments have on the evaluation of retrieval results. Very high correlations were found among the rankings of systems produced using diAerent relevance judgment sets. The high correlations indicate that the comparative evaluation of retrieval performance is stable despite substantial diAerences in relevance judgments, and thus reaArm the use of the TREC collections as laboratory tools. Published by Elsevier Science Ltd.