Software Techniques for Multimodal Input Processing in Realtime Interactive Systems

Multimodal interaction frameworks are an efficient means of utilizing many existing processing and fusion techniques in a wide variety of application areas, even by non-experts. However, the application of these frameworks to highly interactive application areas like VR, AR, MR, and computer games in a reusable, modifiable, and modular manner is not straightforward. It currently lacks some software technical solutions that (1) preserve the general decoupling principle of platforms and at the same time (2) provide the required close temporal as well as semantic coupling of involved software modules and multimodal processing steps. This thesis approches current challenges and aims at providing the research community with a framework that fosters repeatability of scientific achievements and the ability to built on previous results.

[1]  Dennis Wiebusch,et al.  A uniform semantic-based access model for realtime interactive systems , 2014, 2014 IEEE 7th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS).

[2]  Philippe A. Palanque,et al.  Fusion engines for multimodal input: a survey , 2009, ICMI-MLMI '09.

[3]  Marc Erich Latoschik,et al.  Short Paper: Engineering Realtime Interactive Systems: Coupling and Cohesion of Architecture Mechanisms , 2010, EGVE/EuroVR/VEC.

[4]  ISO / IEC 25010 : 2011 Systems and software engineering — Systems and software Quality Requirements and Evaluation ( SQuaRE ) — System and software quality models , 2013 .

[5]  Martin Fischbach,et al.  Engineering Variance: Software Techniques for Scalable, Customizable, and Reusable Multimodal Processing , 2014, HCI.

[6]  Tobias Baur,et al.  The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time , 2013, ACM Multimedia.

[7]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[8]  Wolfgang Wahlster,et al.  User and discourse models for multimodal communication , 1991 .

[9]  Marc Erich Latoschik.  A user interface framework for multimodal VR interactions , 2005, ICMI '05.

[10]  Marc Erich Latoschik,et al.  Simulator X: A scalable and concurrent architecture for intelligent realtime interactive systems , 2011, 2011 IEEE Virtual Reality Conference.

[11]  David B. Koons,et al.  Iconic: speech and depictive gestures at the human-machine interface , 1994, CHI Conference Companion.

[12]  Marc Erich Latoschik,et al.  Utilize speech and gestures to realize natural interaction in a virtual environment , 1998, IECON '98. Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.98CH36200).

[13]  James M. Tainter.  The World-Builder , 1927 .

[14]  Denis Lalanne,et al.  HephaisTK: a toolkit for rapid prototyping of multimodal interfaces , 2009, ICMI-MLMI '09.

[15]  Steven K. Feiner,et al.  Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality , 2003, ICMI '03.

[16]  Kristinn R. Thórisson,et al.  Integrating Simultaneous Input from Speech, Gaze, and Hand Gestures , 1991, AAAI Workshop on Intelligent Multimedia Interfaces.

[17]  Thierry Ganille,et al.  ICARE software components for rapidly developing multimodal interfaces , 2004, ICMI '04.

[18]  Umer Farooq,et al.  API usability: CHI'2009 special interest group meeting , 2009, CHI Extended Abstracts.

[19]  Jean-Yves Lionel Lawson,et al.  The openinterface framework: a tool for multimodal interaction. , 2008, CHI Extended Abstracts.

[20]  Martin Fischbach,et al.  SiXton's curse — Simulator X demonstration , 2011, 2011 IEEE Virtual Reality Conference.

[21]  William Ribarsky,et al.  Evaluation of a multimodal interface for 3D terrain visualization , 2002, IEEE Visualization, 2002. VIS 2002..

[22]  Martin Fischbach,et al.  Evaluating scala, actors, & ontologies for intelligent realtime interactive systems , 2012, VRST '12.

[23]  Michael Johnston,et al.  Finite-state multimodal integration and understanding , 2005, Natural Language Engineering.

[24]  Martin Fischbach,et al.  Fusion of Mixed-Reality Tabletop and Location-Based Applications for Pervasive Games , 2014, ITS '14.

[25]  Dennis Wiebusch,et al.  Decoupling the entity-component-system pattern using semantic traits for reusable realtime interactive systems , 2015, 2015 IEEE 8th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS).

[26]  Gerd Bruder,et al.  Blending Real and Virtual Worlds Using Self-reflection and Fiducials , 2012, ICEC.

[27]  Martin Fischbach,et al.  Exploring multimodal interaction techniques for a mixed reality digital surface , 2014, ISMAR.

[28]  Gerd Bruder,et al.  smARTbox A Portable Setup for Intelligent Interactive Applications , 2012, Mensch & Computer Workshopband.

[29]  Philip R. Cohen,et al.  QuickSet: multimodal interaction for distributed applications , 1997, MULTIMEDIA '97.

[30]  Gerd Bruder,et al.  smARTbox: out-of-the-box technologies for interactive art and exhibition , 2012, VRIC '12.

[31]  C. Y. Thielman,et al.  Natural Language with Integrated Deictic and Graphic Gestures , 1989, HLT.

[32]  Beat Signer,et al.  Mudra: a unified multimodal interaction framework , 2011, ICMI '11.

[33]  Sharon L. Oviatt,et al.  Multimodal Interfaces: A Survey of Principles, Models and Frameworks , 2009, Human Machine Interaction.