Discovering modalities for adaptive multimodal interfaces

mobile terminals, be it cell phones, PDAs, or gaming consoles, the range of features becomes wider. And the way interactive content is presented becomes increasingly diverse, yet the ways to interact with applications, the user-interface capabilities, remain restricted to a small screen, audio input and output, and occasionally a stylus pointing instrument. On the other side, the environment in which mobile communication occurs becomes ever more crowded with devices offering wide ranges of interface modalities. We designed a scheme that when a mobile user comes into the physical range of devices hosting modalities (screens/audio systems, etc), these ambient interface devices are bound into an overall multimodal user interface. Literally, whenever suitable devices are available, the user interface can be tailored to the needs of the user, as well as the requirements of the application. While the principle is rather straightforward, there are a number of associated problems, including: