A scalable comparison-shopping agent for the World-Wide Web

The WorldWideWeb is less agent-friendly than we might hope. Most information on the Web is presented in loosely structured natural language text with no agent-readable semantics. HTML annotations structure the display of Web pages, but provide virtually no insight into their content. Thus, the designers of intelligent Web agents need to address the following questions: (1) To what extent can an agent understand information published at Web sites? (2) Is the agent's understanding sufficient to provide genuinely useful assistance to users? (3) Is site-specific hand-coding necessary, or can the agent automatically extract information from unfamiliar Web sites? (4) What aspects of the Web facilitate this competence? In this paper we investigate these issues with a case study using ShopBot, a fully-implemented, domainindependent comparison-shopping agent. Given the home pages of several online stores, ShopBot autonomously learns how to shop at those vendors. After learning, it is able to speedily visit over a dozen software and CD vendors, extract product information, and summarize the results for the user. Preliminary studies show that ShopBot enables users to both find superior prices and substantially reduce Web shopping time. Remarkably, ShopBot achieves this performance without sophisticated natural language processing, and requires only minimal knowledge about different product domains. Instead, ShopBot relies on a combination of heuristic search, pattern matching, and inductive learning techniques. PERMISSION TO COPY WITHOUT FEE ALL OR OR PART OF THIS MATERIAL IS GRANTED PROVIDED THAT THE COPIES ARE NOT MADE OR DISTRIBUTED FOR DIRECT COMMERCIAL ADVANTAGE, THE ACM copyRIGHT NOTICE AND THE TITLE OF THE PUBLICATION AND ITS DATE APPEAR, AND NOTICE IS GIVEN THAT COPYING IS BY PERMISSION OF ACM. To COPY OTHERWISE, OR TO REPUBLISH, REQUIRES A FEE AND/OR SPECIFIC PERMISSION. AGENTS '97 CONFERENCE PROCEEDINGS, COPYRIGHT 1997 ACM.

[1]  David Chapman,et al.  Pengi: An Implementation of a Theory of Activity , 1987, AAAI.

[2]  Tom M. Mitchell,et al.  Experience with a learning personal assistant , 1994, CACM.

[3]  Henry Lieberman,et al.  Letizia: An Agent That Assists Web Browsing , 1995, IJCAI.

[4]  Jeffrey D. Ullman,et al.  Answering queries using templates with binding patterns (extended abstract) , 1995, PODS '95.

[5]  Ian Horswill,et al.  Cultural Support for Improvisation , 1992, AAAI.

[6]  Craig A. Knoblock,et al.  Retrieving and Integrating Data from Multiple Information Sources , 1993, Int. J. Cooperative Inf. Syst..

[7]  Oren Etzioni,et al.  Category Translation: Learning to Understand Information on the Internet , 1995, IJCAI.

[8]  Ian Horswill,et al.  Analysis of Adaptation and Environment , 1995, Artif. Intell..

[9]  Daniel S. Weld,et al.  Planning to Gather Information , 1996, AAAI/IAAI, Vol. 1.

[10]  Pattie Maes,et al.  Agents that reduce work and information overload , 1994, CACM.

[11]  Pattie Maes,et al.  Learning Interface Agents , 1993, AAAI.

[12]  Donald A. Norman,et al.  How might people interact with agents , 1994, CACM.

[13]  Oren Etzioni,et al.  A softbot-based interface to the Internet , 1994, CACM.

[14]  Jeffrey C. Schlimmer,et al.  Software Agents: Completing Patterns and Constructing User Interfaces , 1993, J. Artif. Intell. Res..

[15]  Daniel S. Weld,et al.  Planning to gather inforrnation , 1996, AAAI 1996.

[16]  Thorsten Joachims,et al.  WebWatcher : A Learning Apprentice for the World Wide Web , 1995 .

[17]  Kristian J. Hammond,et al.  FAQ finder: a case-based approach to knowledge navigation , 1995, Proceedings the 11th Conference on Artificial Intelligence for Applications.

[18]  Tom M. Mitchell,et al.  A Personal Learning Apprentice , 1992, AAAI.

[19]  Anand Rajaraman,et al.  Answering queries using templates with binding patterns (extended abstract) , 1995, PODS.