Computed Ontology-based Situation Awareness of Multi-User Observations

In recent years, we have developed a framework of human-computer interaction that offers recognition of various communication modalities including speech, lip movement, facial expression, handwriting/drawing, gesture, text and visual symbols. The framework allows the rapid construction of a multimodal, multi-device, and multi-user communication system within crisis management. This paper reports the approaches used in multi-user information integration (input fusion) and multimodal presentation (output fission) modules, which can be used in isolation, but also as part of the framework. The latter is able to specify and produce contextsensitive and user-tailored output combining language, speech, visual-language and graphics. These modules provide a communication channel between the system and users with different communication devices. By the employment of ontology, the system’s view about the world is constructed from multi-user observations and appropriate multimodal responses are generated.

[1]  Roger C. Schank,et al.  SCRIPTS, PLANS, GOALS, AND UNDERSTANDING , 1988 .

[2]  L. I. Perlovsky.  Emotions, learning and control , 1999, Proceedings of the 1999 IEEE International Symposium on Intelligent Control Intelligent Systems and Semiotics (Cat. No.99CH37014).

[3]  Armando Fox,et al.  The Interactive Workspaces Project: Experiences with Ubiquitous Computing Rooms , 2002, IEEE Pervasive Comput..

[4]  Zhenke Yang,et al.  Developing Concept-Based User Interface using Icons for Reporting Observations , 2008 .

[5]  Mohammed Yeasin,et al.  Speech-gesture driven multimodal interfaces for crisis management , 2003, Proc. IEEE.

[6]  Marilyn A. Walker,et al.  MATCH: An Architecture for Multimodal Dialogue Systems , 2002, ACL.

[7]  Nl,et al.  Intelligent system for exploring dynamic crisis environments , 2006 .

[8]  Yukiko I. Nakano,et al.  MACK: Media lab Autonomous Conversational Kiosk , 2002 .

[9]  Mirela Popa,et al.  Intelligent system for exploring dynamic crisis environments , 2006 .

[10]  Bob Carpenter,et al.  The logic of typed feature structures , 1992 .

[11]  Léon J. M. Rothkrantz,et al.  A Multimodal Human-Computer Interaction Framework for Research into Crisis Management , 2007 .

[12]  Jesper Kjeldskov,et al.  Interaction Design for Handheld Computers , 2002 .

[13]  Donald A. Norman,et al.  Things that make us smart , 1979 .

[14]  Roberta Catizone,et al.  Multimodal Generation in the COMIC Dialogue System , 2005, ACL.

[15]  Mark Steedman,et al.  Combinatory Categorial Grammar , 2011 .

[16]  Wolfgang Wahlster,et al.  Dialogue Systems Go Multimodal: The SmartKom Experience , 2006, SmartKom.

[17]  Nalini Venkatasubramanian,et al.  CAMAS: a citizen awareness system for crisis mitigation , 2004, SIGMOD '04.

[18]  Sharon L. Oviatt,et al.  When do we interact multimodally?: cognitive load and multimodal communication patterns , 2004, ICMI '04.