Cyberbroker: A lightweight web object collector

Centralized heavyweight collectors have been used to collect text-based web objects for building search engines. However, these collectors do not meet the need of collecting big web objects such as images and videos. We propose a lightweight mobile on-call object collector for the world wide web. Using this approach, we can deploy a collector easily on the site being indexed. We present a prototype system consisting of mobile Java collectors and search engines for collecting HTML/image objects. We provide analysis to find a lower bound in terms of total data transfer time required. This idealized scenario is complemented with another model based on implementation of the HTTP over TCP. The lower bound and the upper bound of the gains accrued are then validated using experiments.