Robots in the Web : threat or treat ?

Robots have been operating in the World Wide Web for over a year. In that time they have performed useful tasks, but also on occasion wreaked havoc in the networks. This paper investigates the advantages and disadvantages of robotsd, with an emphasis on robots used for resource discovery. New alternativrr resource discovery strategies are discussed and compared. It concludes that while current robotx will be useful in the immediate future, they will become less effective and more problematic as the web grows