Optimizing Result Prefetching in Web Search Engines with Segmented Indices

Publisher Summary The sheer size of the WWW and the efforts of search engines to index significant portions of it have caused many search engines to partition their inverted index of the Web into several disjoint segments (partial indices). The partitioning of the index impacts the manner in which the engines process queries. Most engines also use some form of query result caching, where results of queries that were served are cached for some time. In particular, query results may be prefetched in anticipation of user requests. Such scenario occurs when the engine retrieves (for a certain query) more results than will initially be returned to the user. Search engine users have been observed to browse through very few pages of results for queries that they submit. This behavior of users suggests that prefetching many results upon processing an initial query is not efficient, since the user who initiated the search will not request most of the prefetched results. However, a policy that abandons result prefetching in favor of retrieving just the first page of search results might not make optimal use of system resources as well.