The Web is becoming a universal information dissemination medium, due to a number of factors including its support for content dynamicity. A growing number of Web information providers post near real-time updates in domains such as auctions, stock markets, bulletin boards, news, weather, roadway conditions, sports scores, etc. External parties often wish to capture this information for a wide variety of purposes ranging from online data mining to automated synthesis of information from multiple sources. There has been a great deal of work on the design of systems that can process streams of data from Web sources, but little attention has been paid to how to produce these data streams, given that Web pages generally require "pull-based" access.
In this paper we introduce a new general-purpose algorithm for monitoring Web information sources, effectively converting pull-based sources into push-based ones. Our algorithm can be used in conjunction with continuous query systems that assume information is fed into the query engine in a push-based fashion. Ideally, a Web monitoring algorithm for this purpose should achieve two objectives: (1) timeliness and (2) completeness of information captured. However, we demonstrate both analytically and empirically using real-world data that these objectives are fundamentally at odds. When resources available for Web monitoring are limited, and the number of sources to monitor is large, it may be necessary to sacrifice some timeliness to achieve better completeness, or vice versa. To take this fact into account, our algorithm is highly parameterized and targets an application-specified balance between timeliness and completeness. In this paper we formalize the problem of optimizing for a flexible combination of timeliness and completeness, and prove that our parameterized algorithm is a 2- approximation in all cases, and in certain cases is optimal.
[1]
Umberto Bertelè,et al.
Nonserial Dynamic Programming
,
1972
.
[2]
David J. DeWitt,et al.
The Niagara Internet Query System
,
2001,
IEEE Data Eng. Bull..
[3]
Edith Cohen,et al.
Maintaining time-decaying stream aggregates
,
2003,
J. Algorithms.
[4]
Hector Garcia-Molina,et al.
Synchronizing a database to improve freshness
,
2000,
SIGMOD '00.
[5]
Marc Najork,et al.
A large‐scale study of the evolution of Web pages
,
2004,
Softw. Pract. Exp..
[6]
Lili Qiu,et al.
The content and access dynamics of a busy Web site: findings and implications
,
2000
.
[7]
Jenny Edwards,et al.
An adaptive model for optimizing performance of an incremental web crawler
,
2001,
WWW '01.
[8]
Calton Pu,et al.
WebCQ-detecting and delivering information changes on the web
,
2000,
CIKM '00.
[9]
Philip S. Yu,et al.
Optimal crawling strategies for web search engines
,
2002,
WWW '02.
[10]
Wei HanOregon.
Conquer : a Continual Query System for Update Monitoring in the Www
,
1999
.
[11]
Sandeep Pandey,et al.
Monitoring the dynamic web to respond to continuous queries
,
2003,
WWW '03.
[12]
Calton Pu,et al.
Continual Queries for Internet Scale Event-Driven Information Delivery
,
1999,
IEEE Trans. Knowl. Data Eng..
[13]
Toshihide Ibaraki,et al.
Resource allocation problems - algorithmic approaches
,
1988,
MIT Press series in the foundations of computing.