Tools and experiments in multimodal interaction

Author Tommi Ilmonen Title Tools and Experiments in Multimodal Interaction The goal of this study is to explore different strategies for multimodal human-computer interaction. Where traditional human-computer interaction uses a few common user interface metaphors and devices, multimodal interaction seeks new application areas with novel interaction devices and metaphors. Exploration of these new areas involves creation of new application concepts and their implementation. In some cases the interaction mimics human-human interaction while in other cases the interaction model is only loosely tied to the physical world. In the virtual orchestra concept a conductor can conduct a band of virtual musicians. Both the motion and sound of the musicians is synthesized with a computer. A critical task in this interaction is the analysis of the conductor motion and control of the sound synthesis. A system that performs these tasks is presented. The system is also capable of extracting emotional content from the conductor’s motion. While the conductor follower system was originally developed using a commercial motion tracker, an alternative low-cost motion tracking system was also made. The new system used accelerometers with application-specific signal processing for motion capture. One of the basic tasks of the conductor follower and other gesture-based interaction systems is to refine raw user input data into information that is easy to use in the application. For this purpose a new approach was developed: FLexible User Input Design (FLUID). This is a toolkit that simplifies the management of novel interaction devices and offers general-purpose data conversion and analysis algorithms. FLUID was used in a virtual reality drawing applications AnimaLand and Helma. Also new particle system models and a graphics distribution system were developed for these applications. The traditional particle systems were enhanced by adding moving force fields that interact with each other. The interacting force fields make the animations more lively and credible. Graphics distribution become an issue if one wants to render 3D graphics with a cost-effective PC-cluster. A graphics distribution method based on network broadcast was created to minimize the amount of data traffic, thus increasing performance. Many multimodal applications also need a sound synthesis and processing engine. To meet these needs the Mustajuuri toolkit was developed. Mustajuuri is a flexible and efficient sound signal processing framework with support for sound processing in virtual environments.