Evaluating the Effectiveness of Automated Assistance for Web Searching

Web search engines offering personalized automated assistance have the potential to improve the searching process. We report the results of a within subjects, counter balanced empirical evaluation to determine whether automated assistance improves searching performance. Thirty subjects interacted with two fully functional, information retrieval systems. The systems were identical in all respects except that one offered automated assistance while the other did not. The study used a client side automated assistance application, a 2G Text REtrieval Conference document collection and six topics. Results from the empirical evaluation indicate that automated assistance can statistically improve searching performance; however, the improvement is not as dramatic as one might expect, with an approximately 20% performance increase, as measured by number of relevance documents retrieved. We discuss the implications for Web systems and future research.