论文信息 - Characterizing the Web Using a New Uniform Sampling Approach

Characterizing the Web Using a New Uniform Sampling Approach

Web is one the biggest source of information for many. It is also increasingly growing. For easier use of the Web, Web search engines (WSEs) are being used frequently. However, there is little information about the characteristics of the Web and also WSEs. One usual way to analysis these characteristics is to use a uniform sample. In such approaches, instead of working on the entire Web we can work on a small subset of the Web representing entire Web. In this paper, we propose a new method, called bucket-based sampling (BBS), to gather this small but uniform subset of the Web. The analyses show that BBS improves the samples' uniformity, at least 6.95% respecting PAGERANK-SMP, one of the best existing methods. Using samples gathered by BBS, we compare the relative size of seven famous WSEs. We also estimate some important characteristics of the Web. For example we estimate that the size of indexable Web is around 20.14 billion pages.

Ali Movaghar-Rahimabadi | Hamid Mousavi | Mohammad Ebrahim Rafiei

[1] Marc Najork,et al. On near-uniform URL sampling , 2000, Comput. Networks.

[2] Krishna Bharat,et al. Who links to whom: mining linkage between Web sites , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[3] Andrei Z. Broder,et al. Graph structure in the Web , 2000, Comput. Networks.

[4] Jon M. Kleinberg,et al. The Web as a Graph: Measurements, Models, and Methods , 1999, COCOON.

[5] Rajeev Motwani,et al. The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[6] Edward T. O'Neill,et al. A Methodology for Sampling the World Wide Web , 2001 .

[7] Richard J. Lipton,et al. Random walks, universal traversal sequences, and the complexity of maze problems , 1979, 20th Annual Symposium on Foundations of Computer Science (sfcs 1979).

[8] David M. Pennock,et al. Methods for Sampling Pages Uniformly from the World Wide Web , 2001 .

[9] G. Clark,et al. Reference , 2008 .

[10] Andrei Z. Broder,et al. A Technique for Measuring the Relative Size and Overlap of Public Web Search Engines , 1998, Comput. Networks.

[11] R. Armstrong. Review of Research. , 1990 .

[12] James H. Lambert,et al. A Methodology for , 2000 .

[13] L. Asz. Random Walks on Graphs: a Survey , 2022 .

[14] Steve Chien,et al. Approximating Aggregate Queries about Web Pages via Random Walks , 2000, VLDB.

[15] Antonio Gulli,et al. The indexable web is more than 11.5 billion pages , 2005, WWW '05.