Determination of the Consistency of Relevance Judgments and the Reliability of Search Strategies among Information Specialists for the Aerospace Materials Information Center.

Abstract : The ability of various AMIC information specialists to prepare search strategies for document retrieval was studied by providing ten typical search request statements to seven information specialists. Each specialist prepared search strategies independently. Significant variations occurred among the strategies, although even with these variations, reasonably consistent document returns resulted. The experiment indicated that the proper interpretation of a search request and the conversion of the request to an appropriate search strategy even with a well-established thesaurus is a considerably more difficult task than indexing. In another experiment, the consistency of relevance judgments among information specialists was examined. Relevance judgment is more difficult than indexing and search strategy preparation. (Author)