A Novel Architecture of a Parallel Web Crawler

ABSTRACT Due to the explosion in the size of the WWW[1,4,5] it becomes essential to make the crawling process parallel. In this paper we present an architecture for a parallel crawler that consists of multiple crawling processes called as C-procs which can run on network of workstations. The proposed crawler is scalable, is resilient against system crashes and other event. The aim of this architecture is to efficiently and effectively crawl the current set of publically indexable web pages so that we can maximize the download rate while minimizing the overhead from parallelization Keywords WWW, Search Engines, Crawlers, Parallel Crawlers. 1. INTRODUCTION The World-Wide Web has undergone explosive, exponential growth. As a consequence, users find themselves unable to browse the ever-changing, distributed hyperlink structure of the web. Furthermore, they are subjected to information overload – information is too abundant . With the increasing number of information resources on the Web, it is often more difficult to locate the resources that are relevant to a given need. Henceforth, Search Engines[2,3] are becoming equally important tool in locating relevant information.