A Toolkit for Multimodal Interface Design: An Empirical Investigation

This paper introduces a comparative multi-group study carried out to investigate the use of multimodal interaction metaphors (visual, oral, and aural) for improving learnability (or usability from first time use) of interface-design environments. An initial survey was used for taking views about the effectiveness and satisfaction of employing speech and speech-recognition for solving some of the common usability problems. Then, the investigation was done empirically by testing the usability parameters: efficiency, effectiveness, and satisfaction of three design-toolkits (TVOID, OFVOID, and MMID) built especially for the study. TVOID and OFVOID interacted with the user visually only using typical and time-saving interaction metaphors. The third environment MMID added another modality through vocal and aural interaction. The results showed that the use of vocal commands and the mouse concurrently for completing tasks from first time use was more efficient and more effective than the use of visual-only interaction metaphors.

[1]  Chih-Kai Chang,et al.  A language for developing collaborative learning activities on World Wide Web , 1996, Proceedings of 20th International Computer Software and Applications Conference: COMPSAC '96.

[2]  Dimitris I. Rigas,et al.  Multimedia e-mail data browsing: the synergistic use of various forms of auditory stimuli , 2003, Proceedings ITCC 2003. International Conference on Information Technology: Coding and Computing.

[3]  Dimitrios Rigas,et al.  Experiments in using structured musical sound, synthesised speech and environmental stimuli to communicate information: is there a case for integration and synergy? , 2001, Proceedings of 2001 International Symposium on Intelligent Multimedia, Video and Speech Processing. ISIMP 2001 (IEEE Cat. No.01EX489).

[4]  Stephen J. Payne,et al.  Task-Action Grammars: A Model of the Mental Representation of Task Languages , 1986, Hum. Comput. Interact..

[5]  Murray Crease,et al.  GUIDELINES FOR AUDIO-ENHANCEMENT OF GRAPHICAL USER INTERFACE WIDGETS , 2002 .

[6]  Stephen A. Brewster,et al.  Web-based haptic applications for blind people to create virtual graphs , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[7]  Helen Petrie,et al.  Authoring hypermedia systems for blind people , 1995 .

[8]  T. V. Raman.  The Audio Desktop , 1997 .

[9]  Ian Oakley,et al.  Guidelines for the Design of Haptic Widgets , 2002 .

[10]  K. C. Persaud,et al.  Blind subjects construct conscious mental images of visual scenes encoded in musical form , 2000, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[11]  Ian Oakley,et al.  Putting the feel in ’look and feel‘ , 2000, CHI.

[12]  K. N. Bateson,et al.  CAL programs developed in advanced programming environments for teaching electrical engineering , 1994 .

[13]  Shi-Kuo Chang,et al.  A visual editor for multimedia application development , 2002, Proceedings 22nd International Conference on Distributed Computing Systems Workshops.

[14]  Dimitrios Rigas,et al.  Guidelines for auditory interface design : an empirical investigation , 1996 .

[15]  Joseph Finkelstein,et al.  CO-ED: a development platform for interactive patient education , 2002, International Conference on Computers in Education, 2002. Proceedings..

[16]  Stéphane Conversy,et al.  Auditory illusions for audio feedback , 1996, CHI Conference Companion.

[17]  T. V. Raman.  Emacspeak-an audio desktop , 1997, Proceedings IEEE COMPCON 97. Digest of Papers.

[18]  Stephen A. Brewster,et al.  The design and evaluation of a sonically enhanced tool palette , 2005, TAP.