论文信息 - The Research of Web Page De-duplication Based on Web Pages Reshipment Statement

The Research of Web Page De-duplication Based on Web Pages Reshipment Statement

Web page de-duplication module is an important part of search engine system, which can improve its performance and quality with filtering the web pages downloaded by crawler system of search engine and eliminating the duplicated web pages. This paper from the source of duplicated web pages--reshipment proposes a web page de-duplication method that the information including original websites and web titles are extracted to eliminate duplicated web pages based on feature codes. Experiments show that this method can achieve satisfactory results in eliminating large-scale duplicated web pages.

Dong-sheng Liu | Min-yan Wang

[1] Chen Xiao-zhi. Algorithm of Parallelized Elimination of Duplicated Web Pages Based on Map/Reduce , 2007 .

[2] Ding Zhen-Guo,et al. Research of large-scale URL Filter Base on Bloom Filter , 2008 .

[3] Gao Kai. The Strategy on Processing Replicated Web Collections , 2006 .

[4] Li Xiao-Ming,et al. Two Effective Functions on Hashing URL , 2004 .

[5] Jin-yan Chen. Finding near replicas of Web pages based on Fourier transform: Finding near replicas of Web pages based on Fourier transform , 2008 .

[6] Daniel P. Lopresti,et al. Models and algorithms for duplicate document detection , 1999, Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cat. No.PR00318).

[7] Zhang Ya-ping. Finding near replicas of Web pages based on Fourier transform , 2008 .

[8] Pierre Baldi,et al. Modeling the Internet and the Web: Probabilistic Methods and Algorithms: Baldi/Probabilistic , 2002 .

[9] Pierre Baldi,et al. Modeling the Internet and the Web: Probabilistic Methods and Algorithms. By Pierre Baldi, Paolo Frasconi, Padhraic Smith, John Wiley and Sons Ltd., West Sussex, England, 2003. 285 pp ISBN 0 470 84906 1 , 2006, Inf. Process. Manag..