Finding near-replicas of documents on the Web
暂无分享,去创建一个
We consider how to efficiently compute the overlap between all pairs of web documents. This information can be used to improve web crawlers, web archivers and in the presentation of search results, among others. We report statistics on how common replication is on the web, and on the cost of computing the above information for a relatively large subset of the web - about 24 million web pages which corresponds to about 150 Gigabytes of textual information.