CRAYSE: design and implementation of efficient text search algorithm in a web crawler

CRAYSE1 is a SEarch WHIle CRAwl application, intended to perform fast searching of text in web pages. A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner. This process is also called spidering. Search engines, use spidering as a means of providing up-to-date data. Most of the existing web-crawlers archive the contents of the web starting from the input URL. Search engines index the results of web-crawlers and then perform searching when queried. As such, the searching is not performed while crawling. Hence such softwares can not be used for general use by web browsers. Also, the existing search mechanism in web browsers, search only on the current page and not recursively through all the links present in that page. In order to overcome such disadvantages, we propose in this paper to implement a web crawler that searches for a pattern efficiently and recursively through all the links including pdf links while crawling. CRAYSE can be used as a general purpose open source software by web browsers. It can also be used for offine searching. Further, the applications that require selective archival of web pages (based on the presence of a key word), can deploy CRAYSE for efficient search operations. This paper focusses on the design and implementation of CRAYSE and its demonstration through web applications.