A goal-oriented user interface for personalized semantic search

Users have high-level goals when they browse the Web or perform searches. However, the two primary user interfaces positioned between users and the Web, Web browsers and search engines, have very little interest in users' goals. Present-day Web browsers provide only a thin interface between users and the Web, and present-day search engines rely solely on keyword matching. This thesis leverages large knowledge bases of semantic information to provide users with a goal-oriented Web browsing experience. By understanding the meaning of Web pages and search queries, this thesis demonstrates how Web browsers and search engines can proactively suggest content and services to users that are both contextually relevant and personalized. This thesis presents (1) Creo, a Programming by Example system that allows users to teach their computers how to automate interactions with their favorite Web sites by providing a single demonstration, (2) Miro, a Data Detector that matches the content of a Web page to high-level user goals, and allows users to perform semantic searches, and (3) Adeo, an application that streamlines browsing the Web on mobile devices, allowing users to complete actions with a minimal amount of input and output. An evaluation with 34 subjects found that they were more effective at completing tasks when using these applications, and that the subjects would use these applications if they had access to them. Beyond these three user interfaces, this thesis also explores a number of underlying issues, including (1) automatically providing semantics to unstructured text, (2) building robust applications on top of messy knowledge bases, (3) leveraging surrounding context to disambiguate concepts that have multiple meanings, and (4) learning new knowledge by reading the Web. Thesis Supervisor: Henry Lieberman Title: Research Scientist A Goal-Oriented User Interface for Personalized Semantic Search by Alexander James Faaborg

[1]  H. Lieberman,et al.  Common Sense on the Go : Giving Mobile Applications an Understanding of Everyday Life , 2004 .

[2]  John V. Carlis,et al.  End-user Web Automation: Challenges, Experiences, Recommendations , 2001, World Conference on the WWW and Internet.

[3]  Henry Lieberman,et al.  A model of textual affect sensing using real-world knowledge , 2003, IUI '03.

[4]  John Domingue,et al.  Magpie: supporting browsing and navigation on the semantic web , 2004, IUI '04.

[5]  Erik T. Mueller,et al.  Open Mind Common Sense: Knowledge Acquisition from the General Public , 2002, OTM.

[6]  Push Singh,et al.  Common Sense Conversations: Understanding Casual Conversation using a Common Sense Database , 2003 .

[7]  Leonardo Bonanni,et al.  CounterIntelligence: Augmented Reality Kitchen , 2005 .

[8]  David Garlan,et al.  Lightweight structure in text , 2002 .

[9]  Pedro M. Domingos,et al.  Personalizing web sites for mobile users , 2001, WWW '01.

[10]  Gregory D. Abowd,et al.  CyberDesk: a framework for providing self-integrating context-aware services , 1998, IUI '98.

[11]  Bonnie A. Nardi,et al.  Collaborative, programmable intelligent agents , 1998, CACM.

[12]  Henry Lieberman,et al.  GOOSE: A Goal-Oriented Search Engine with Commonsense , 2002, AH.

[13]  Leo Obrst,et al.  The Semantic Web: A Guide to the Future of XML, Web Services and Knowledge Management , 2003 .

[14]  Michael Bolin.  End-user programming for the Web , 2005 .

[15]  Earl J. Wagner,et al.  End-User Debugging for Electronic Commerce , 2003 .

[16]  H. Lieberman,et al.  Agents for the User Interface , 2003 .

[17]  Jakob Nielsen,et al.  Ten Usability Heuristics , 2006 .

[18]  Mathias Bauer,et al.  TrIAs: trainable information assistants for cooperative problem solving , 1999, AGENTS '99.

[19]  Xinyu Hugo Liu,et al.  Semantic understanding and commonsense reasoning in an adaptive photo agent , 2002 .

[20]  Earl Joseph Wagner.  Woodstein : a Web interface agent for debugging e-commerce , 2003 .

[21]  Push Singh,et al.  Context Sensing using Speech and Common Sense , 2004, Workshop On Spoken Language Understanding For Conversational Systems And Higher Level Linguistic Information For Speech Processing.

[22]  Push Singh,et al.  LifeNet: A Propositional Model of Ordinary Human Activity , 2003 .

[23]  Mathias Bauer,et al.  InfoBeams—configuration of personalized information assistants , 1998, IUI '99.

[24]  Peter H. Alesso,et al.  Developing semantic web services , 2004 .

[25]  Ashwani Kumar Bache,et al.  i-Seek : An Intelligent System for Eliciting and Explaining Knowledge , 2005 .

[26]  Push Singh,et al.  The Public Acquisition of Commonsense Knowledge , 2002 .

[27]  Mathias Bauer,et al.  Instructible information agents for Web mining , 2000, IUI '00.

[28]  David R. Karger,et al.  Piggy Bank: Experience the Semantic Web inside your web browser , 2005, J. Web Semant..

[29]  Brad A. Myers,et al.  Creating Dynamic World Wide Web Pages By Demonstration , 1997 .

[30]  Pedro M. Domingos,et al.  Version Space Algebra and its Application to Programming by Demonstration , 2000, ICML.

[31]  Henry Lieberman,et al.  Adaptive Linking between Text and Photos Using Common Sense Reasoning , 2002, AH.

[32]  Frank van Harmelen,et al.  A semantic web primer , 2004 .

[33]  Alex Safonov.  Web macros by example: users managing the WWW of applications , 1999, CHI EA '99.

[34]  Marti A. Hearst.  Automatic Acquisition of Hyponyms from Large Text Corpora , 1992, COLING.

[35]  Henry Lieberman,et al.  Robust Photo Retrieval Using World Semantics , 2002 .

[36]  Henry Lieberman,et al.  Grammex: defining grammars by example , 1998, CHI Conference Summary.

[37]  Milind S. Pandit,et al.  The selection recognition agent: instant access to relevant information and operations , 1997, IUI '97.

[38]  Enrico Motta,et al.  Magpie - Towards a Semantic Web Browser , 2003, SEMWEB.

[39]  Barbara Barry,et al.  Beating Some Common Sense into Interactive Applications , 2003 .

[40]  Carl Lagoze,et al.  Semantic Browsing , 2003, ECDL.

[41]  I. V. Ramakrishnan,et al.  Browsing fatigue in handhelds: semantic bookmarking spells relief , 2005, WWW '05.

[42]  Juliana Freire,et al.  Automating Web navigation with the WebVCR , 2000, Comput. Networks.

[43]  Ramanathan V. Guha,et al.  Contexts for the Semantic Web , 2004, SEMWEB.

[44]  Atsushi Sugiura,et al.  Internet scrapbook: automating Web browsing tasks by demonstration , 1998, UIST '98.

[45]  Henry Lieberman,et al.  Training Agents to Recognize Text by Example , 1999, AGENTS '99.

[46]  Henry Lieberman,et al.  Integrating user interface agents with conventional applications , 1998, IUI '98.

[47]  Thomas Bonura,et al.  From documents to objects: an overview of LiveDoc , 1998, SGCH.

[48]  Thomas Bonura,et al.  Drop zones: an extension to LiveDoc , 1998, SGCH.

[49]  Nicholas Kushmerick,et al.  Wrapper induction: Efficiency and expressiveness , 2000, Artif. Intell..

[50]  Juliana Freire,et al.  WebViews: accessing personalized web content and services , 2001, WWW '01.

[51]  Chris Schmandt,et al.  Autonomous interactive intermediaries: social intelligence for mobile communication agents , 2005 .

[52]  Thomas B. Passin,et al.  Explorer's guide to the semantic web , 2004 .

[53]  Hugo Liu,et al.  ConceptNet — A Practical Commonsense Reasoning Tool-Kit , 2004 .

[54]  Alan Cooper,et al.  About Face: The Essentials of User Interface Design , 1995 .

[55]  Henry Lieberman,et al.  A commonsense approach to predictive text entry , 2004, CHI EA '04.

[56]  D. A. Quan,et al.  How to make a semantic web browser , 2004, WWW '04.

[57]  Hugo Liu,et al.  Teaching Machines about Everyday Life , 2004 .

[58]  Wei-Ying Ma,et al.  Detecting web page structure for adaptive viewing on small form factor devices , 2003, WWW '03.

[59]  Pedro M. Domingos,et al.  Mixed initiative interfaces for learning tasks: SMARTedit talks back , 2001, IUI '01.

[60]  Mathias Bauer,et al.  Programming by example: programming by demonstration for information agents , 2000, CACM.

[61]  Espinosa Christlieb,et al.  Reducing complexity of consumer electronics interfaces using commonsense reasoning , 2005 .

[62]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[63]  Henry Lieberman,et al.  How to wreck a nice beach you sing calm incense , 2005, IUI.

[64]  Henry Lieberman,et al.  Common sense investing: bridging the gap between expert and novice , 2004, CHI EA '04.

[65]  Ramanathan V. Guha,et al.  TAP: a Semantic Web platform , 2003, Comput. Networks.

[66]  Henry Lieberman,et al.  Visualizing the affective structure of a text document , 2003, CHI Extended Abstracts.

[67]  Henry Lieberman,et al.  Autonomous interface agents , 1997, CHI.

[68]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[69]  Ramanathan V. Guha,et al.  Semantic search , 2003, WWW '03.

[70]  Henry Lieberman,et al.  Exploring the Web with reconnaissance agents , 2001, Commun. ACM.

[71]  Push Singh,et al.  The Open Mind Common Sense project , 2002 .