Multimodal Architectures: Issues and Experiences

The penetration of mobile device in western countries is still increasing The Italian case is really surprising: every single Italian has more than one mobile terminal Thus, considering this large potential audience, there is real need for innovation and new services In this context, usable multimodal services could have an unexpected impact on social behaviour Nevertheless, the research community should be able to propose a framework for building generic multi-modal services, covering all their lifecycle We are currently defining an architecture for building coordinated simultaneous multimodal applications trying to use as much as possible open source software: our goal is to define a set of tools for enabling a rapid deployment of a generic multimodal service In our opinion, a platform based on open source software could meet the expectations of a large numbers of service developers A special effort for enabling a mass diffusion of mobile multimodal services should be focused on the client side, where the situation is still evolving.

[1]  Michelle X. Zhou,et al.  A probabilistic approach to reference resolution in multimodal user interfaces , 2004, IUI '04.

[2]  Yacine Bellik,et al.  Multimodal output specification / simulation platform , 2005, ICMI '05.

[3]  Sharon L. Oviatt,et al.  Mutual disambiguation of recognition errors in a multimodel architecture , 1999, CHI '99.

[4]  Sharon Oviatt,et al.  Multimodal interactive maps: designing for human performance , 1997 .

[5]  Nils Klarlund,et al.  Towards SMIL as a foundation for multimodal, multimedia applications , 2001, INTERSPEECH.

[6]  Franca Garzotto,et al.  Designing model hypermedia applications , 1997, HYPERTEXT '97.

[7]  Sharon L. Oviatt,et al.  Multimodal Integration - A Statistical View , 1999, IEEE Trans. Multim..

[8]  Arjeh M. Cohen,et al.  Synchronized Multimedia Integration Language (SMIL) 2.0 , 1998 .

[9]  Mary Zajicek,et al.  Proceedings of the 2003 conference on Universal usability , 2003 .

[10]  Ivan Marsic,et al.  A framework for rapid development of multimodal interfaces , 2003, ICMI '03.

[11]  Philip R. Cohen,et al.  MULTIMODAL INTERFACES THAT PROCESS WHAT COMES NATURALLY , 2000 .

[12]  Rainer Simon,et al.  Tool-supported single authoring for device independence and multimodality , 2005, Mobile HCI.

[13]  E. A. den Os,et al.  MUST - Multimodal and multilingual services for small mobile terminals , 2002 .

[14]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[15]  Sharon L. Oviatt,et al.  Perceptual user interfaces: multimodal interfaces that process what comes naturally , 2000, CACM.

[16]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[17]  Joëlle Coutaz,et al.  Two Case Studies of Software Architecture for Multimodal Interactive Systems: VoicePaint and a Voice-enabled Graphical Notebook , 1992, Engineering for Human-Computer Interaction.

[18]  Michael Strube,et al.  Architecture and implementation of multimodal plug and play , 2003, ICMI '03.

[19]  Sharon L. Oviatt,et al.  Ten myths of multimodal interaction , 1999, Commun. ACM.

[20]  Gregg C. Vanderheiden,et al.  Abstract user interface representations: how well do they support universal access? , 2002 .

[21]  Jukka Riekki,et al.  Middleware support for implementing context-aware multimodal user interfaces , 2004, MUM '04.

[22]  Jesper Ø. Olsen.  ICSLP'98 : Proceedings of the 5th International Conference on Spoken Language Processing, November 30-December 4, 1998, Sydney, Australia , 1998 .

[23]  Michael Johnston,et al.  Multimodal language processing , 1998, ICSLP.

[24]  Trung Bui,et al.  Multimodal Dialogue Management - State of the art , 2006 .

[25]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[26]  Sharon L. Oviatt,et al.  Taming recognition errors with a multimodal interface , 2000, CACM.

[27]  Johanna D. Moore,et al.  Proceedings of the Conference on Human Factors in Computing Systems , 1989 .