Scalability Bottlenecks of the CiteSeerX Digial Library Search Engine

As the document collection and user population increase, the capability and performance of a digital library such as CiteSeerX maybe limited by some bottlenecks. This paper describes the current infrastructure of the CiteSeerX academic digital library search engine, outlines its current bottlenecks and proposes feasible solutions. These bottlenecks exist in various components of the system including hardware, web crawling, text extraction and storage. The hardware bottleneck is the increasing difficulty to maintain a cluster consisting of almost twenty physical servers. The solution is to merge some servers and implement the whole system under a virtual architecture. The web crawling bottleneck is that the seed URLs are biased on on computer science, information sciences, technology and related fields. One of the approaches to balance the domain distribution of our crawl repository, is to obtain seed URLs from generic search engines. Another bottleneck is the average time to extract text from the crawled documents. To reduce the processing time, we have proposed a new extraction model using message queues and multiple threads. Preliminary experiments indicates that the average time to extract a document can be reduced by an order of magnitude. The storage bottleneck is that as the data repository size grows, a better tool is required to manage the storage, transferring, sharing and backing up of our files. Hadoop provides a promising tool to parallelize data analysis and the Hadoop File System provides a solution for shared storage. All solutions to the current bottlenecks are either under testing or on our roadmap. Our indexing protocol does not have foreseeable bottlenecks in the near future.