Automatically generating high-quality user interfaces for appliances

In this dissertation, I show that many appliance usability problems can be addressed by moving the user interface from the appliance to a handheld device that the user is already carrying, such as a Personal Digital Assistant (PDA) or mobile phone. This approach, called the Personal Universal Controller (PUC), takes advantage of the increasing pervasiveness of wireless communication technologies that will allow handheld devices to communicate directly with appliances. Automatic generation of the appliance user interface allows the PUC to create interfaces that are customized to the platform of the controller device, the user's previous experience, and all the appliances that are present in the user's current environment. This dissertation makes several contributions to the state of the art in automatic interface generation: (1) Automatic generation that makes use of dependency information to determine a better structure for the generated user interfaces; (2) The general Smart Templates technique for incorporating domain-specific design conventions into an appliance specification and automatically rendering the conventions appropriately on different platforms and in different interface modalities; (3) Algorithms that apply knowledge of similarity between specifications and interfaces the user has previously seen to generate new interfaces that are personally consistent; (4) Algorithms that use a model of the content flow between appliances to generate task-based interfaces that combine functionality from multiple appliances. An evaluation of the PUC system compared the automatically generated interfaces for two all-in-one printers with the manufacturer's interfaces for the same two appliances and found that users of the automatically generated interfaces were twice as fast and four times as successful for both common and complex tasks. The evaluation also shows that the PUC's consistency features allow users to be twice as fast when using a new appliance that is similar to an appliance they have previously encountered. This evaluation is the first known user study of automatically generated interfaces compared to human designs, and it shows that automatic generation of user interfaces for end users is now viable for interactive systems.

[1]  Daniela Gorski Trevisan,et al.  The Continuity Property in Mixed Reality and Multiplatform Systems: a Comparative Study , 2004, CADUI.

[2]  Mari Ostendorf,et al.  SABLE: a standard for TTS markup , 1998, ICSLP.

[3]  Krzysztof Z. Gajos,et al.  SUPPLE: automatically generating user interfaces , 2004, IUI '04.

[4]  Brad A. Myers Using hand-held devices and pcs together , 2001 .

[5]  Jonathan Grudin,et al.  The case against user interface consistency , 1989, CACM.

[6]  Jana Koehler,et al.  Web Service Composition - Current Solutions and Open Problems , 2003 .

[7]  Fabio Paternò,et al.  Design and development of multidevice user interfaces through multiple logical descriptions , 2004, IEEE Transactions on Software Engineering.

[8]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[9]  Jeffrey Nichols,et al.  Generating Consistent Interfaces for Appliances , 2005 .

[10]  Phyllis Reisner,et al.  What is inconsistency? , 1990, INTERACT.

[11]  Fabio Paternò,et al.  CTTE: Support for Developing and Analyzing Task Models for Interactive System Design , 2002, IEEE Trans. Software Eng..

[12]  Steffen Staab,et al.  Web Services: Been There, Done That? , 2003, IEEE Intell. Syst..

[13]  Maddy D. Brouwer-Janse,et al.  Interfaces for consumer products: “how to camouflage the computer?” , 1992, CHI '92.

[14]  James D. Foley,et al.  Coupling application design and user interface design , 1992, CHI.

[15]  Pedro A. Szekely,et al.  Design alternatives for user interface management sytems based on experience with COUSIN , 1985, CHI '85.

[16]  David E. Kieras,et al.  The GOMS family of user interface analysis techniques: comparison and contrast , 1996, TCHI.

[17]  Mark W. Newman,et al.  User interfaces when and where they are needed: an infrastructure for recombinant computing , 2002, UIST '02.

[18]  Belur V. Dasarathy,et al.  Nearest neighbor (NN) norms: NN pattern classification techniques , 1991 .

[19]  Dan R. Olsen,et al.  A Programming Language Basis for User Interface Management , 1989 .

[20]  Guruduth Banavar,et al.  Tooling and system support for authoring multi-device applications , 2004, J. Syst. Softw..

[21]  Fabio Paternò,et al.  ConcurTaskTrees: A Diagrammatic Notation for Specifying Task Models , 1997, INTERACT.

[22]  Brad A. Myers,et al.  Automatic, look-and-feel independent dialog creation for graphical user interfaces , 1990, CHI '90.

[23]  Laurent Karsenty,et al.  Inter‐Usability of Multi‐Device Systems – A Conceptual Framework , 2005 .

[24]  Pat Hanrahan,et al.  ICrafter: A Service Framework for Ubiquitous Computing Environments , 2001, UbiComp.

[25]  Terry Winograd,et al.  Bringing Design to Software , 1996 .

[26]  Jeffrey Nichols,et al.  Report on the INCITS/V2 AIAP-URC Standard , 2004 .

[27]  Ping Luo,et al.  Beyond interface builders: model-based interface tools , 1993, INTERCHI.

[28]  E KierasDavid,et al.  The GOMS family of user interface analysis techniques , 1996 .

[29]  Prasun Dewan,et al.  Comparing end-user and intelligent remote control interface generation , 2005, Personal and Ubiquitous Computing.

[30]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[31]  Charles Wiecha,et al.  ITS: a tool for rapidly developing interactive applications , 1990, TOIS.

[32]  Dan R. Olsen,et al.  Cross-modal interaction using XWeb , 2000, UIST '00.

[33]  Dieter Schmalstieg,et al.  Agents that talk and hit back: animated agents in augmented reality , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[34]  W. Buxton Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[35]  Marc Abrams,et al.  UIML: An Appliance-Independent XML User Interface Language , 1999, Comput. Networks.

[36]  Erhard Rahm,et al.  Similarity flooding: a versatile graph matching algorithm and its application to schema matching , 2002, Proceedings 18th International Conference on Data Engineering.

[37]  Lorne Olfman,et al.  User Interface Consistency Across End-User Applications: The Effects on Mental Models , 1998, J. Manag. Inf. Syst..

[38]  Jeffrey Nichols,et al.  Improving automatic interface generation with smart templates , 2004, IUI '04.

[39]  Nick Hammond,et al.  Consistency and Compatibility in Human-Computer Dialogue , 1981, Int. J. Man Mach. Stud..

[40]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[41]  Ronald Rosenfeld,et al.  A universal speech interface for appliances , 2004, INTERSPEECH.

[42]  Erhard Rahm,et al.  Generic Schema Matching with Cupid , 2001, VLDB.

[43]  Jacob Eisenstein,et al.  XIML: a common representation for interaction data , 2002, IUI '02.

[44]  Ping Luo,et al.  Facilitating the exploration of interface design alternatives: the HUMANOID model of interface design , 1992, CHI.

[45]  Alexander I. Rudnicky,et al.  Universal speech interfaces , 2001, INTR.

[46]  Jeffrey Nichols,et al.  Generating remote control interfaces for complex appliances , 2002, UIST '02.

[47]  Pedro A. Szekely Retrospective and Challenges for Model-Based Interface Development , 1996, DSV-IS.

[48]  Kenneth R. Koedinger,et al.  Predictive human performance modeling made easy , 2004, CHI.

[49]  Wayne H. Ward,et al.  The CMU Air Travel Information Service: Understanding Spontaneous Speech , 1990, HLT.

[50]  Jérôme Euzenat,et al.  A Survey of Schema-Based Matching Approaches , 2005, J. Data Semant..

[51]  Alexander I. Rudnicky,et al.  A unified design for human-machine voice interaction , 2001, CHI Extended Abstracts.

[52]  G. Engelbeck,et al.  A test of a common elements theory of transfer , 1986, CHI '86.

[53]  Benjamin Michotte,et al.  USIXML: A Language Supporting Multi-path Development of User Interfaces , 2004, EHCI/DS-VIS.

[54]  Krzysztof Z. Gajos,et al.  Automatically generating custom user interfaces for users with physical disabilities , 2006, Assets '06.

[55]  Marc Abrams,et al.  Building Multi-Platform User Interfaces with UIML , 2001, CADUI.

[56]  Angel R. Puerta,et al.  A Model-Based Interface Development Environment , 1997, IEEE Softw..

[57]  Jeffrey Nichols,et al.  Studying the use of handhelds to control smart appliances , 2003, 23rd International Conference on Distributed Computing Systems Workshops, 2003. Proceedings..

[58]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[59]  Ronald Rosenfeld,et al.  Speech Graffiti vs. Natural Language: Assessing the User Experience , 2004, HLT-NAACL.

[60]  Randy H. Katz,et al.  Composable ad-hoc mobile services for universal interaction , 1997, MobiCom '97.

[61]  Pedro M. Domingos,et al.  Reconciling schemas of disparate data sources: a machine-learning approach , 2001, SIGMOD '01.

[62]  Daniel S. Weld,et al.  Cross-Device Consistency in Automatically Generated User Interfaces , 2004 .

[63]  Jean Vanderdonckt Knowledge-Based Systems for Automated User Inter- face Generation : the TRIDENT Experience , 1995 .

[64]  Ben Shneiderman,et al.  Visual and Textual Consistency Checking Tools for Graphical User Interfaces , 1997, IEEE Trans. Software Eng..

[65]  Krzysztof Z. Gajos,et al.  Ligature: Gesture-Based Configuration of the E21 Intelligent Environment , 2001 .

[66]  Wendy A. Kellogg,et al.  CONCEPTUAL CONSISTENCY IN THE USER INTERFACE: EFFECTS ON USER PERFORMANCE , 1987 .

[67]  Stina Nylander,et al.  The Ubiquitous Interactor - Device Independent Access to Mobile Services , 2004, CADUI.

[68]  Marc Spraragen,et al.  An intelligent assistant for interactive workflow composition , 2004, IUI '04.

[69]  James D. Foley,et al.  Providing high-level control and expert assistance in the user interface presentation design , 1993, INTERCHI.

[70]  Henry Lieberman,et al.  A goal-oriented interface to consumer electronics using planning and commonsense reasoning , 2006, IUI '06.

[71]  Markus Chimani,et al.  DiamondHelp : A Graphical User Interface Framework for Human-Computer Collaboration , 2004 .

[72]  Fausto Giunchiglia,et al.  Encoding classifications into lightweight ontologies , 2007 .

[73]  Anthony J. Hornof,et al.  GLEAN: a computer-based tool for rapid GOMS model usability evaluation of user interface designs , 1995, UIST '95.

[74]  James D. Foley,et al.  Coupling a UI framework with automatic generation of context-sensitive animated help , 1990, UIST '90.

[75]  James D. Foley,et al.  A second generation user interface design environment: the model and the runtime architecture , 1993, INTERCHI.

[76]  Stefan Edlund,et al.  A Universal Information Appliance , 1999, IBM Syst. J..

[77]  Richard S. Hall,et al.  Gravity: supporting dynamically available services in client-side applications , 2003, ESEC/FSE-11.

[78]  Jean Vanderdonckt Advice-giving systems for selecting interaction objects , 1999, Proceedings User Interfaces to Data Intensive Systems.

[79]  Jeffrey Nichols,et al.  Requirements for automatically generating multi-modal interfaces for complex appliances , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[80]  Patrick Baudisch,et al.  Flat volume control: improving usability by hiding the volume control hierarchy in the user interface , 2004, CHI.

[81]  Christine Halverson,et al.  An authoring technology for multidevice Web applications , 2004, IEEE Pervasive Computing.

[82]  Brad A. Myers,et al.  Past, Present and Future of User Interface Software Tools , 2000, TCHI.

[83]  Raymonde Guindon,et al.  Cognitive science and its applications for human-computer interaction , 1988 .

[84]  John Rheinfrank,et al.  Design languages , 1996 .

[85]  Pablo Castells,et al.  Declarative interface models for user interface construction tools: the MASTERMIND approach , 1995, EHCI.

[86]  Jean Vanderdonckt,et al.  Applying model-based techniques to the development of UIs for mobile computers , 2001, IUI '01.

[87]  B. Schneiderman,et al.  Designing the User Interface. Strategies for Effective Human-Computer Interaction , 1992 .

[88]  Krzysztof Z. Gajos,et al.  Automatically personalizing user interfaces , 2003, IJCAI 2003.

[89]  Krzysztof Z. Gajos,et al.  Fast and Robust Interface Generation for Ubiquitous Applications , 2005, UbiComp.

[90]  Anupriya Ankolekar,et al.  Automated discovery, interaction and composition of Semantic Web services , 2003, J. Web Semant..