论文信息 - Crawler-Friendly Web Servers

Crawler-Friendly Web Servers

In this paper we study how to make web servers (e.g., Apache) more crawler friendly. Current web servers offer the same interface to crawlers and regular web surfers, even though crawlers and surfers have very different performance requirements. We evaluate simple and easy-to-incorporate modifications to web servers so that there are significant bandwidth savings. Specifically, we propose that web servers export meta-data archives decribing their content.

[1] Zhen Liu,et al. Optimal Robot Scheduling for Web Search Engines , 1998 .

[2] Press Niso. Information Retrieval Application Service Definition and Protocol Specification for Open Systems Interconnection, Z39.50-1995 , 1994 .

[3] Eric J. Miller. An Introduction to the Resource Description Framework. , 1998 .

[4] Dan Brickley,et al. Resource Description Framework (RDF) Model and Syntax Specification , 2002 .

[5] V. Jacobson,et al. Congestion avoidance and control , 1988, CCRV.

[6] Hector Garcia-Molina,et al. Synchronizing a database to improve freshness , 2000, SIGMOD 2000.

[7] Jeffrey C. Mogul,et al. Improving HTTP Latency , 1995, Comput. Networks ISDN Syst..

[8] Luis Gravano,et al. STARTS: Stanford Protocol Proposal for Internet Retrieval and Search , 1997 .

[9] Anja Feldmann,et al. Potential benefits of delta encoding and data compression for HTTP , 1997, SIGCOMM '97.

[10] Paul Barford,et al. Generating representative Web workloads for network and server performance evaluation , 1998, SIGMETRICS '98/PERFORMANCE '98.

[11] C. Lee Giles,et al. Accessibility of information on the web , 1999, Nature.