A dynamic reconfiguration model for a distributed web crawling system

A web crawling system employing a parallel and distributed architecture needs to have a mechanism to bring the whole system in a coordinated state when the nodes are added to or removed from the system. This paper presents an efficient dynamic reconfiguration model that can be used in such a system. The study shows that this model leads to some nice properties, such as load balance and low traffic in the system, which contribute to high performance. Currently this model is being implemented in WebGather, a well-known Chinese and English web search engine.