A New Mercator Web Crawler

Mercator is a scalable web crawler which crawl the user request for a corresponding documents and fetch it with the URL request as a response and send to the user in an efficient way. We enumerate the major components of scalable web crawler, comment on alternatives, extensibility, customizability and tradeoffs in their design. This paper describes the architecture of the Mercator. In this paper we are propose an efficient way to handle the request of the user without direct contact with Rewind Input Stream (RIS) through protocols modules, by this response time will decrease on RIS in such manner that it will be more efficient. We modified the architecture of the Mercator as a web crawler by adding a new RequestResponse proxy server (RRS)component.