Ensuring Retrieval Effectiveness in Distributed Digital Libraries

Abstract We find that dissemination of collection-wide information (CWI) in a distributed collection of documents is needed to achieve retrieval effectiveness comparable to that of a centralized collection. Complete dissemination is unnecessary. The required dissemination level depends upon the content-skew of the distributed collection, i.e., how documents are allocated among sites. Low dissemination is needed for random document allocation, but higher levels are needed when documents are allocated on the basis of their content. We define parameters to control dissemination and document allocation and present results from four document collections. These results provide insight into the necessary technology underlying digital libraries. We also describe the architecture of the Networked Computer Science Technical Report Library (NCSTRL), a concrete example of a system that fits our model of a distributed archive.