Scalability and reliability in a distributed search engine

We have developed a distributed search engine, called cooperative search engine (CSE), in order to retrieve fresh information. In CSE, a local search engine located in each Web server makes an index of local pages. A meta search server integrates these local search engines in order to realize a global search engine. In such a way, the communication delay occurs at retrieval time. So, it is thought to be difficult to search quickly. However we have developed several speedup techniques in order to realize real time retrieval. In addition, the meta server is a single point of failure in CSE. So, we propose redundancy of meta search servers in order to increase availability of CSE. In this paper we describe scalability and reliability of CSE and their evaluations.

[1]  Jim Fullton,et al.  Architecture of the Whois++ Index Service , 1996, RFC.

[2]  Peter B. Danzig,et al.  The Harvest Information Discovery and Access System , 1995, Comput. Networks ISDN Syst..

[3]  Kenneth P. Birman,et al.  The process group approach to reliable distributed computing , 1992, CACM.

[4]  Minoru Uehara,et al.  Persistent cache in Cooperative Search Engine , 2002, Proceedings 22nd International Conference on Distributed Computing Systems Workshops.

[5]  Minoru Uehara.  Fault tolerant computing in computational field model , 1997, Proceedings International Conference and Workshop on Engineering of Computer-Based Systems.

[6]  Minoru Uehara,et al.  Distributed information retrieval by using cooperative meta search engines , 2001, Proceedings 21st International Conference on Distributed Computing Systems Workshops.