Context-aware intelligent user interfaces for supporting system use

Two trends can be observed in current applications: the amount of functionality offered is constantly increasing and applications are more often used in mobile settings. Both trends often lead to a decrease in the usability of applications. This effect can be countered by Intelligent User Interfaces that facilitate the interaction between user and application. In order to support the user in an optimal way, the Intelligent User Interface has to be aware of the user’s needs and adapt the provided support to her current situation, i.e. her current context. The relevant context information can thereby be gathered from the environment (e.g. her current location) or from sophisticated user models that reflect the user’s behavior. In this thesis, we present a novel approach of combining user and environmental context information for supporting the system use. As context information is often very error-prone and the user‘s workflow should not be disrupted by erroneous interaction support, we adapt the presentation of the support to the reliability of the context information. Therefore, we use different levels of proactivity from unobtrusive highlighting to automatically performing tasks. The presented approach -called AUGUR- is application independent and is able to support the interaction for arbitrary existing applications, even across application boundaries. For that purpose, we developed a novel application modeling language that is able to model applications and their relationships to context. The application models can thereby be defined by the application developer, learned by AUGUR, and augmented by the end-user. The interaction can be facilitated by an Intelligent User Interface in two different ways: on the one hand by supporting the entering of data and on the other hand by simplifying the navigation within and between applications. For supporting the user in entering data, we contribute three approaches based on (i) the user’s previous interactions, (ii) the information represented in the application models, and (iii) the semantics of the data required by the user interface and the context information currently relevant for the user. For the latter, we developed a novel algorithm that combines string-based and semantic similarity measures. AUGUR supports the user’s navigation in three different ways: It can (i) guide the user through an application, (ii) provide navigation shortcuts to other applications, and (iii) reduce the user interface to the most relevant functionality for mobile use. For guiding the user, we developed a novel algorithm called FxL that is able to predict the next relevant interaction element. For the interface adaptation, we introduce a novel approach based on FxL to determine the elements that should be presented to the user for mobile use according to her current situation. We realized the developed concepts of AUGUR in a working prototype. Further, we evaluated the usability of the context-aware support provided by AUGUR in a user study, and showed that it can significantly increase the usability of an application.

[1]  George Buchanan,et al.  Improving mobile internet usability , 2001, WWW '01.

[2]  Saul Greenberg,et al.  USING UNIX: COLLECTED TRACES OF 168 USERS , 1988 .

[3]  Jaideep Srivastava,et al.  Selecting the right interestingness measure for association patterns , 2002, KDD.

[4]  Yannis Kalfoglou,et al.  Ontology mapping: the state of the art , 2003, The Knowledge Engineering Review.

[5]  Melanie Hartmann,et al.  AUGUR: Interface Adaptation for Small Screen Devices , 2009, Advances in Ubiquitous User Modelling.

[6]  Deborah L. McGuinness,et al.  Toward establishing trust in adaptive agents , 2008, IUI '08.

[7]  Melanie Hartmann,et al.  Challenges in Developing User-Adaptive Intelligent User Interfaces , 2009, LWA.

[8]  Thomas G. Dietterich,et al.  TaskTracer: a desktop environment to support multi-tasking knowledge workers , 2005, IUI.

[9]  Anthony Jameson,et al.  Modelling both the Context and the User , 2001, Personal and Ubiquitous Computing.

[10]  Brad A. Myers,et al.  Enabling rich human-agent interaction for a calendar scheduling agent , 2005, CHI EA '05.

[11]  David Poole,et al.  Predicting Future User Actions by Observing Unmodified Applications , 2000, AAAI/IAAI.

[12]  Iryna Gurevych,et al.  What to be? - Electronic Career Guidance Based on Semantic Relatedness , 2007, ACL.

[13]  Joanna McGrenere,et al.  Impact of screen size on performance, awareness, and user satisfaction with adaptive graphical user interfaces , 2008, CHI.

[14]  Russell Beale,et al.  Architectures to make simple visualisations using simple systems , 2000, AVI '00.

[15]  Wolfgang Wahlster,et al.  Distinguishing User Models From Discourse Models , 1988, CL.

[16]  Kristina Höök,et al.  Steps to take before intelligent user interfaces become real , 2000, Interact. Comput..

[17]  Candace L. Sidner,et al.  COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction , 2001, AI Mag..

[18]  Rainer Malaka.  Intelligent User Interfaces for Ubiquitous Computing , 2008, Handbook of Research on Ubiquitous Computing Technology for Real Time Enterprises.

[19]  Iryna Gurevych,et al.  Extracting Lexical Semantic Knowledge from Wikipedia and Wiktionary , 2008, LREC.

[20]  Anthony Jameson,et al.  Pros and Cons of Controllability: An Empirical Study , 2002, AH.

[21]  Constantine Stephanidis,et al.  The PALIO Framework for Adaptive Information Services , 2005 .

[22]  Max Mühlhäuser,et al.  Using Similarity Measures for Context-Aware User Interfaces , 2008, 2008 IEEE International Conference on Semantic Computing.

[23]  Jean Vanderdonckt,et al.  USIXML: A User Interface Description Language Supporting Multiple Levels of Independence , 2004, ICWE Workshops.

[24]  Guanling Chen,et al.  A Survey of Context-Aware Mobile Computing Research , 2000 .

[25]  Dan R. Olsen,et al.  Cross-modal interaction using XWeb , 2000, UIST '00.

[26]  Helmut Schmidt,et al.  Probabilistic part-of-speech tagging using decision trees , 1994 .

[27]  Max Mühlhäuser,et al.  Web based evaluation of proactive multimodal user interfaces , 2007 .

[28]  Hendrik Blockeel,et al.  Sequence prediction with mixed order Markov chains , 2002 .

[29]  Clement T. Yu,et al.  An interactive clustering-based approach to integrating source query interfaces on the deep Web , 2004, SIGMOD '04.

[30]  Tomás E. Uribe,et al.  Deploying a personalized time management agent , 2006, AAMAS '06.

[31]  Gregory D. Abowd,et al.  CyberDesk: a framework for providing self-integrating ubiquitous software services , 1997, UIST '97.

[32]  Max Mühlhäuser,et al.  Tailoring the Interface to Individual Users , 2008 .

[33]  Pattie Maes,et al.  A learning interface agent for scheduling meetings , 1993, IUI '93.

[34]  Stephen F. Smith,et al.  CMRadar: A Personal Assistant Agent for Calendar Management , 2004, AAAI.

[35]  F. Paterno,et al.  Engineering task models , 1997, Proceedings. Third IEEE International Conference on Engineering of Complex Computer Systems (Cat. No.97TB100168).

[36]  Mahadev Satyanarayanan,et al.  Pervasive computing: vision and challenges , 2001, IEEE Wirel. Commun..

[37]  Analía Amandi,et al.  Connecting web applications with interface agents , 2004, Int. J. Web Eng. Technol..

[38]  Vesa Kyllönen,et al.  Context management for end user development of context-aware applications , 2005, MDM '05.

[39]  Max Mühlhäuser,et al.  MundoCore: A light-weight infrastructure for pervasive computing , 2007, Pervasive Mob. Comput..

[40]  Max Mühlhäuser,et al.  Context-Aware Form Filling for Web Applications , 2009, 2009 IEEE International Conference on Semantic Computing.

[41]  Jean Vanderdonckt,et al.  Distributed User Interfaces in Ambient Environment , 2007, AmI Workshops.

[42]  Mary Czerwinski,et al.  Effective Notification Systems Depend on User Trust , 2001, INTERACT.

[43]  Max Mühlhäuser,et al.  Interacting with Context , 2009, QuaCon.

[44]  John Zimmerman,et al.  Vio: a mixed-initiative approach to learning and automating procedural update tasks , 2007, CHI.

[45]  Analía Amandi,et al.  Automatic task model generation for Interface Agent development , 2005, Inteligencia Artif..

[46]  Wolfgang Wahlster,et al.  Readings in Intelligent User Interfaces , 1998 .

[47]  L. A. Hermens,et al.  A machine-learning apprentice for the completion of repetitive forms , 1994, IEEE Expert.

[48]  Ramakrishnan Srikant,et al.  Fast Algorithms for Mining Association Rules in Large Databases , 1994, VLDB.

[49]  Sriram Raghavan,et al.  Crawling the Hidden Web , 2001, VLDB.

[50]  Diane J. Cook,et al.  Online Sequential Prediction via Incremental Parsing: The Active LeZi Algorithm , 2007, IEEE Intelligent Systems.

[51]  Hugo Liu,et al.  ConceptNet — A Practical Commonsense Reasoning Tool-Kit , 2004 .

[52]  Geert Wets,et al.  Defining interestingness for association rules , 2003 .

[53]  Cristina Conati,et al.  Supporting interface customization using a mixed-initiative approach , 2007, IUI '07.

[54]  A. Jameson.  Adaptive interfaces and agents , 2002 .

[55]  Thomas G. Dietterich,et al.  Fewer clicks and less frustration: reducing the cost of reaching the right folder , 2006, IUI '06.

[56]  James D. Foley,et al.  Built-In User Modelling Support, Adaptive Interfaces, and Adaptive Help in UIDE , 1992 .

[57]  Jean Vanderdonckt,et al.  A Unifying Reference Framework for multi-target user interfaces , 2003, Interact. Comput..

[58]  Jacob Eisenstein,et al.  XIML: a common representation for interaction data , 2002, IUI '02.

[59]  Daniel S. Weld,et al.  Exploring the design space for adaptive graphical user interfaces , 2006, AVI '06.

[60]  Ingrid Zukerman,et al.  Bayesian Models for Keyhole Plan Recognition in an Adventure Game , 2004, User Modeling and User-Adapted Interaction.

[61]  Jean Vanderdonckt,et al.  A Review of XML-compliant User Interface Description Languages , 2003, DSV-IS.

[62]  Kevin Chen-Chuan Chang,et al.  Understanding Web query interfaces: best-effort parsing with hidden syntax , 2004, SIGMOD '04.

[63]  Jadwiga Indulska,et al.  XCML: A Runtime Representation for the Context Modelling Language , 2007, Fifth Annual IEEE International Conference on Pervasive Computing and Communications Workshops (PerComW'07).

[64]  William E. Winkler,et al.  AN APPLICATION OF THE FELLEGI-SUNTER MODEL OF RECORD LINKAGE TO THE 1990 U.S. DECENNIAL CENSUS , 1987 .

[65]  Frank Linton,et al.  OWL: A Recommender System for Organization-Wide Learning , 2000, J. Educ. Technol. Soc..

[66]  Matthias Baldauf,et al.  A survey on context-aware systems , 2007, Int. J. Ad Hoc Ubiquitous Comput..

[67]  Allen Cypher,et al.  EAGER: programming repetitive tasks by example , 1991, CHI.

[68]  Marc Abrams,et al.  UIML: An Appliance-Independent XML User Interface Language , 1999, Comput. Networks.

[69]  Fabio Paternò,et al.  Authoring pervasive multimodal user interfaces , 2008, Int. J. Web Eng. Technol..

[70]  Sara Foresti,et al.  Overview of the State of the Art , 2011 .

[71]  Takeshi Chusho,et al.  Automatic filling in a form by an agent for web applications , 2002, Ninth Asia-Pacific Software Engineering Conference, 2002..

[72]  Russell C. Swan,et al.  Perceived usability and system complexity , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[73]  Evgeniy Gabrilovich,et al.  Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis , 2007, IJCAI.

[74]  Anind K. Dey,et al.  Is Context-Aware Computing Taking Control away from the User? Three Levels of Interactivity Examined , 2003, UbiComp.

[75]  Pattie Maes,et al.  Collaborative Interface Agents , 1994, AAAI.

[76]  Thomas G. Dietterich,et al.  Predicting User Tasks : I Know What You ’ re Doing ! , 2005 .

[77]  Max Mühlhäuser,et al.  Web based evaluation of proactive user interfaces , 2008, Journal on Multimodal User Interfaces.

[78]  Bill N. Schilit,et al.  Context-aware computing applications , 1994, Workshop on Mobile Computing Systems and Applications.

[79]  Anind K. Dey,et al.  Support for context-aware intelligibility and control , 2009, CHI.

[80]  Brian D. Davison,et al.  Predicting Sequences of User Actions , 1998 .

[81]  Charles Elkan,et al.  The Field Matching Problem: Algorithms and Applications , 1996, KDD.

[82]  Melanie Hartmann,et al.  Task Models for Proactive Web Applications , 2007, WEBIST.

[83]  Axel Küpper,et al.  Quality of Context: What It Is And Why We Need It , 2004 .

[84]  Markus Chimani,et al.  DiamondHelp: a new interaction design for networked home appliances , 2006, Personal and Ubiquitous Computing.

[85]  Judy Kay,et al.  Learner Control , 2001, User Modeling and User-Adapted Interaction.

[86]  Candace L. Sidner,et al.  COLLAGEN: when agents collaborate with people , 1997, AGENTS '97.

[87]  Matthew A. Jaro,et al.  Probabilistic linkage of large public health data files. , 1995, Statistics in medicine.

[88]  Judy Kay,et al.  The justified user model: A viewable explained user model , 2003 .

[89]  Melanie Hartmann,et al.  Prediction Algorithms for User Actions , 2007, LWA.

[90]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[91]  W. Keith Edwards,et al.  Intelligibility and Accountability: Human Considerations in Context-Aware Systems , 2001, Hum. Comput. Interact..

[92]  Tom M. Mitchell,et al.  A Personal Learning Apprentice , 1992, AAAI.

[93]  Ludger Schmidt,et al.  Antizipative Modellierung des Benutzerverhaltens mit Hilfe von Aktionsvorhersage-Algorithmen , 2004, MMI Interakt..

[94]  Tom Rodden,et al.  "Playing with the Bits" User-Configuration of Ubiquitous Domestic Environments , 2003, UbiComp.

[95]  Pattie Maes,et al.  Agents that reduce work and information overload , 1994, CACM.

[96]  Henry Lieberman,et al.  A goal-oriented web browser , 2006, CHI.

[97]  Anind K. Dey,et al.  Understanding and Using Context , 2001, Personal and Ubiquitous Computing.

[98]  Brad A. Myers,et al.  Citrine: providing intelligent copy-and-paste , 2004, UIST '04.

[99]  Timothy Sohn,et al.  iCAP: Interactive Prototyping of Context-Aware Applications , 2006, Pervasive.

[100]  Eser Kandogan,et al.  Koala: capture, share, automate, personalize business processes on the web , 2007, CHI.

[101]  Andreas Paepcke,et al.  Efficient Web form entry on PDAs , 2001, WWW '01.

[102]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[103]  Clement T. Yu,et al.  Automatic integration of Web search interfaces with WISE-Integrator , 2004, The VLDB Journal.

[104]  Max Mühlhäuser,et al.  Designing and Implementing Smart Spaces , 2007 .

[105]  Melanie Hartmann,et al.  Proactively Adapting Interfaces to Individual Users for Mobile Devices , 2008, AH.

[106]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[107]  Kristina Höök,et al.  A glass box approach to adaptive hypermedia , 1996, User Modeling and User-Adapted Interaction.

[108]  Melanie Hartmann,et al.  Context Models and Context Awareness , 2008, Handbook of Research on Ubiquitous Computing Technology for Real Time Enterprises.

[109]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[110]  Lutz Heuser,et al.  Device Independent Web Applications - The Author Once - Display Everywhere Approach , 2004, ICWE.

[111]  Erhard Rahm,et al.  A survey of approaches to automatic schema matching , 2001, The VLDB Journal.

[112]  Donald A. Norman,et al.  How might people interact with agents , 1994, CACM.

[113]  Henry Lieberman,et al.  Beating Common Sense into Interactive Applications , 2004, AI Mag..

[114]  Kathy Ryall,et al.  Learning hierarchical task models by defining and refining examples , 2001, K-CAP '01.

[115]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[116]  Claudia Linnhoff-Popien,et al.  A Context Modeling Survey , 2004 .

[117]  Krzysztof Z. Gajos,et al.  SUPPLE: automatically generating user interfaces , 2004, IUI '04.

[118]  Nicu Sebe,et al.  Affective multimodal human-computer interaction , 2005, ACM Multimedia.

[119]  Jon Orwant,et al.  Heterogeneous learning in the Doppelgänger user modeling system , 2005, User Modeling and User-Adapted Interaction.

[120]  Andy Hopper,et al.  The active badge location system , 1992, TOIS.

[121]  William W. Cohen,et al.  A Comparison of String Metrics for Matching Names and Records , 2003 .

[122]  Barry Smyth,et al.  The Plight of the Navigator: Solving the Navigation Problem for Wireless Portals , 2002, AH.

[123]  Alexander De Luca,et al.  Automatic form filling on mobile devices , 2008, Pervasive Mob. Comput..

[124]  Jeffrey Nichols,et al.  Generating remote control interfaces for complex appliances , 2002, UIST '02.

[125]  Susan Bull,et al.  An Intelligent Learning Environment with an Open Learner Model for the Desktop PC and Pocket PC , 2003 .