The effect of input mode on inactivity and interaction times of multimodal systems

In this paper, the efficiency and usage patterns of input modes in multimodal dialogue systems is investigated for both desktop and personal digital assistant (PDA) working environments. For this purpose a form-filling travel reservation application is evaluated that combines the speech and visual modalities; three multimodal modes of interaction are implemented, namely: "Click-To-Talk", "Open-Mike" and "Modality-Selection". The three multimodal systems are evaluated and compared with the "GUI-Only" and "Speech-Only" unimodal systems. Mode and duration statistics are computed for each system, for each turn and for each attribute in the form. Turn time is decomposed in interaction and inactivity time and the statistics for each input modeare computed. Results show that multimodal and adaptive interfaces are superior in terms of interaction time, but not always in terms of inactivity time. Also users tend to use themost efficient input mode, although our experiments show abias towards the speech modality.

[1]  A. Potamianos,et al.  Modality tracking in the multimodal Bell Labs Communicator , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).

[2]  Eric Fosler-Lussier,et al.  Information Seeking Spoken Dialogue Systems— Part II: Multimodal Dialogue , 2007, IEEE Transactions on Multimedia.

[3]  Alexandros Potamianos,et al.  BLENDING SPEECH AND VISUAL INPUT IN MULTIMODAL DIALOGUE SYSTEMS , 2006, 2006 IEEE Spoken Language Technology Workshop.

[4]  Hong-Kwang Jeff Kuo,et al.  Dialogue management in the Bell Labs communicator system , 2000, INTERSPEECH.

[5]  Philip R. Cohen,et al.  The role of voice in human-machine communication , 1994 .

[6]  Nicole Yankelovich,et al.  Conversational speech interfaces , 2002 .

[7]  David S. Ebert,et al.  The integrality of speech in multimodal interfaces , 1998, TCHI.

[8]  Sharon L. Oviatt,et al.  The efficiency of multimodal interaction: a case study , 1998, ICSLP.

[9]  P R Cohen,et al.  The role of voice input for human-machine communication. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Chin-Hui Lee,et al.  DESIGN PRINCIPLES AND TOOLS FOR MULTIMODAL DIALOG SYSTEMS , 2000 .

[11]  Emiel Krahmer,et al.  Preferred modalities in dialogue systems , 2000, INTERSPEECH.

[12]  Niels Ole Bernsen,et al.  Is speech the right thing for your application? , 1998, ICSLP.