Collaborative Multimodality

This essay is a personal reflection from an Artificial Intelligence (AI) perspective on the term HCI. Especially for the transfer of AI-based HCI into industrial environments, we survey existing approaches and examine how AI helps to solve fundamental problems of HCI technology. The user and the system must have a collaborative goal. The concept of collaborative multimodality could serve as the missing link between traditional HCI and intuitive human-centred designs in the form of, e.g., natural language interfaces or intelligent environments. Examples are provided in the medical imaging domain.

[1]  T. Metin Sezgin,et al.  Intelligent user interfaces , 2012, 2012 20th Signal Processing and Communications Applications Conference (SIU).

[2]  Manuel Möller,et al.  RadSem: Semantic Annotation and Retrieval for Medical Images , 2009, ESWC.

[3]  Norbert Reithinger,et al.  SmartWeb Handheld - Multimodal Interaction with Ontological Knowledge Bases and Semantic Web Services , 2007, Artifical Intelligence for Human Computing.

[4]  H. Kirchmann,et al.  SmartKom : Symmetric Multimodality in an Adaptive and Reusable Dialogue Shell , 2003 .

[5]  Gobinda G. Chowdhury,et al.  Spinning the Semantic Web: Bringing the World Wide Web to Its Full Potential , 2004 .

[6]  Randy Goebel,et al.  The expansion continues: Stitching together the breadth of disciplines impinging on Artificial Intelligence , 2011, Artif. Intell..

[7]  Randall Davis,et al.  LADDER, a sketching language for user interface developers , 2005, Comput. Graph..

[8]  Harry Bunt,et al.  Multidimensional Dialogue Management , 2006, SIGDIAL Workshop.

[9]  Norbert Reithinger,et al.  An integration framework for a mobile multimodal dialogue system accessing the semantic web , 2005, INTERSPEECH.

[10]  Krzysztof Z. Gajos,et al.  Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces , 2008, CHI.

[11]  Ashwani Kumar,et al.  Miamm — A Multimodal Dialogue System Using Haptics , 2005 .

[12]  Booncharoen Sirinaovakul,et al.  Introduction to the Special Issue , 2002, Comput. Intell..

[13]  James C. Lester,et al.  Modeling and evaluating empathy in embodied companion agents , 2007, Int. J. Hum. Comput. Stud..

[14]  Anthony Jameson,et al.  Introduction to the Special Issue on "Usable AI" , 2009, AI Mag..

[15]  Daniel Sonntag,et al.  Ontologies and Adaptivity in Dialogue for Question Answering , 2010, Studies on the Semantic Web.

[16]  van Hgw Hans Dam,et al.  Dialogue acts in GUIs , 2000 .

[17]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[18]  Wolfgang Wahlster,et al.  Intelligent Interactive Entertainment Grand Challenges , 2006, IEEE Intelligent Systems.

[19]  Peter Norvig,et al.  Artificial intelligence - a modern approach, 2nd Edition , 2003, Prentice Hall series in artificial intelligence.

[20]  Charles Rich,et al.  A Non-Modal Approach to Integrating Dialogue and Action , 2010, AIIDE.

[21]  Norbert Reithinger,et al.  A Discourse and Dialogue Infrastructure for Industrial Dissemination , 2010, IWSDS.

[22]  Candace L. Sidner,et al.  COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction , 2001, AI Mag..

[23]  Henry Lieberman,et al.  User Interface Goals, AI Opportunities , 2009, AI Mag..

[24]  Wolfgang Wahlster,et al.  Fusion and Coordination for Multimodal Interactive Information Presentation , 2005 .

[25]  Sebastian Rudolph,et al.  Foundations of Semantic Web Technologies , 2009 .

[26]  G. Bower,et al.  THE PSYCHOLOGY OF LEARNING AND M·OTIVATION , 2001 .

[27]  Wolfgang Wahlster,et al.  Verbmobil: Foundations of Speech-to-Speech Translation , 2000, Artificial Intelligence.

[28]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[29]  Stephanie Rosenthal,et al.  Using Interaction to Improve Intelligence : How Intelligent Systems Should Ask Users for Input , 2022 .

[30]  Donna M. Johnson,et al.  Politeness: Some Universals in Language Usage (Studies in Interactional Sociolinguistics 4) , 1988 .

[31]  W. Bradford Paley,et al.  Interface and Mind — A “Paper Lecture” about a Domain-Specific Design Methodology based on Contemporary Mind ScienceBenutzerschnittstelle und Verstand — Über eine Design-Methodik basierend auf aktuellen Erkenntnissen der Psychologie , 2009, it Inf. Technol..

[32]  Harry Bunt,et al.  Cooperative Multimodal Communication , 2001, Lecture Notes in Computer Science.