Distributed Indexing of the Web Using Migrating Crawlers

Due to the tremendous increase rate and the high change frequency of Web documents, maintaining an up-to-date index for searching purposes (search engines) is becoming a challenge. The traditional crawling methods are no longer able to catch up with the constantly updating and growing Web. Realizing the problem, in this paper we suggest an alternative distributed crawling method with the use of mobile agents. Our goal is a scalable crawling scheme that minimizes network utilization, keeps up with document changes, employs time realization, and is easily upgradeable.

[1]  Christian F. Tschudin,et al.  Towards mobile cryptography , 1998, Proceedings. 1998 IEEE Symposium on Security and Privacy (Cat. No.98CB36186).

[2]  Roy T. Fielding,et al.  Hypertext Transfer Protocol - HTTP/1.0 , 1996, RFC.

[3]  Akihiko Ohsuga,et al.  Security for Mobile Agents , 2000, AOSE.

[4]  Nicholas Carriero,et al.  Linda and Friends , 1986, Computer.

[5]  Peter B. Danzig,et al.  The Harvest Information Discovery and Access System , 1995, Comput. Networks ISDN Syst..

[6]  N. Asokan,et al.  Protecting the computation results of free-roaming agents , 2005, Personal Technologies.

[7]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[8]  Peter B. Danzig,et al.  Harvest: A Scalable, Customizable Discovery and Access System , 1994 .

[9]  Joachim Hammer,et al.  Using mobile crawlers to search the Web efficiently , 2000, ACIS Int. J. Comput. Inf. Sci..

[10]  Hector Garcia-Molina,et al.  Parallel crawlers , 2002, WWW.

[11]  Vijay Varadharajan.  Security enhanced mobile agents , 2000, CCS.

[12]  C. Lee Giles,et al.  Accessibility of information on the web , 1999, Nature.

[13]  Marco Gori,et al.  Focused Crawling Using Context Graphs , 2000, VLDB.

[14]  Marc Najork,et al.  Mercator: A scalable, extensible Web crawler , 1999, World Wide Web.

[15]  Soumen Chakrabarti,et al.  Accelerated focused crawling through online relevance feedback , 2002, WWW.

[16]  Aaron Kershenbaum,et al.  Mobile Agents: Are They a Good Idea? , 1996, Mobile Object Systems.

[17]  Joachim Hammer,et al.  Using the Web Efficiently: Mobile Crawlers , 1998 .

[18]  Martin van den Berg,et al.  Focused Crawling: A New Approach to Topic-Specific Web Resource Discovery , 1999, Comput. Networks.