Using critical path analysis to model multimodal human-computer interaction

In this paper, the concept of multimodal human-computer interaction is explored. It is proposed that multimodality can be defined from human or technology perspectives, which place emphasis on different attributes of the system. Furthermore, in this paper it is argued that the most effective definition of multimodality concentrates on task and goal dependencies. Not only does this permit consideration over and above the technology/human distinction, but also allows consideration of multiple tasks. In order to explore this notion, critical path analysis is used to develop models of multimodal systems. The model describes multimodal HCI, and allows consideration of the effects of modality dependency. The models allow prediction of transaction time under various conditions. Predictions arising from these models are shown to be good fits with data obtained from user trials. Thus, it is proposed that one can develop and evaluate preliminary versions of multimodal systems prior to prototype development.

[1]  Alan W. Biermann,et al.  A Voice- and Touch-Driven Natural Language Editor and its Performance , 1992, Int. J. Man Mach. Stud..

[2]  Richard A. Bolt,et al.  The human interface: Where people and computers meet , 1984 .

[3]  Sanjay Manandhar,et al.  Observations on using speech input for window navigation , 1990, INTERACT.

[4]  John E. Laird,et al.  Stimulus-Response Compatibility , 1986 .

[5]  R Schweickert,et al.  Critical-path scheduling of mental processes in a dual task. , 1980, Science.

[6]  Gary M. Olson,et al.  The growth of cognitive modeling in human-computer interaction since GOMS , 1990 .

[7]  Philip R. Cohen,et al.  Synergistic use of direct manipulation and natural language , 1989, CHI '89.

[8]  Akio Ando,et al.  Speech timing prediction in multimodal human-computer interaction , 1997, INTERACT.

[9]  Bonnie E. John,et al.  Toward an Engineering Model of Stimulus-Response Compatibility , 1990 .

[10]  Günther Gediga,et al.  A critique and empirical investigation of the “One-Best-Way-Models” in human-computer interaction , 1988 .

[11]  Richard A. Bolt,et al.  Multi-modal natural dialogue , 1992, CHI '92.

[12]  Michael E. Atwood,et al.  Project Ernestine: Validating a GOMS Analysis for Predicting and Explaining Real-World Task Performance , 1993, Hum. Comput. Interact..

[13]  William A. Ainsworth.  Optimization of String Length for Spoken Digit Input with Error Correction , 1988, Int. J. Man Mach. Stud..

[14]  Alexander G. Hauptmann,et al.  Gestures with Speech for Graphic Manipulation , 1993, Int. J. Man Mach. Stud..

[15]  Chris Baber,et al.  Modelling Error Recovery and Repair in Automatic Speech Recognition , 1993, Int. J. Man Mach. Stud..

[16]  Richard Schweickert,et al.  A critical path generalization of the additive factor method: Analysis of a stroop task , 1978 .

[17]  Antonella De Angeli,et al.  Integration and synchronization of input modes during multimodal human-computer interaction , 1997, CHI.

[18]  Chris Baber,et al.  Modelling the effects of constraint upon speech-based human-computer interaction , 1999, Int. J. Hum. Comput. Stud..

[19]  B H Kantowitz,et al.  Double stimulation with varying response requirements. , 1974, Journal of experimental psychology.

[20]  Michael E. Atwood,et al.  Social interaction in the use and design of a workstation: two contexts of interaction , 1995 .

[21]  S. Kicha Ganapathy,et al.  A synthetic visual environment with hand gesturing and voice input , 1989, CHI '89.

[22]  P. Thomas.  The social and interactional dimensions of human-computer interfaces , 1995 .

[23]  Dan Diaper.  Human–Computer Interaction , 2003 .

[24]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[25]  Jan Noyes,et al.  Workload and the use of automatic speech recognition: The effects of time and resource demands , 1996, Speech Commun..

[26]  Paul Milgram,et al.  On Validating Human Performance Simulation Models , 1991 .

[27]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[28]  Gale Martin,et al.  The Utility of Speech Input in User-Computer Interfaces , 1989, Int. J. Man Mach. Stud..

[29]  S. Mackenzie,et al.  A comparison of input device in elemental pointing and dragging task , 1991, CHI 1991.

[30]  K. G. Lockyer,et al.  Critical Path Analysis and Other Project Network Techniques , 1984 .

[31]  Allen Newell,et al.  The keystroke-level model for user performance time with interactive systems , 1980, CACM.

[32]  Abigail Sellen,et al.  A comparison of input devices in element pointing and dragging tasks , 1991, CHI.

[33]  R. Schweickert.  Latent network theory: scheduling of processes in sentence verification and the Stroop effect. , 1983, Journal of experimental psychology. Learning, memory, and cognition.

[34]  David E. Kieras,et al.  An Approach to the Formal Analysis of User Complexity , 1999, Int. J. Man Mach. Stud..

[35]  A Ehrenstein,et al.  Scheduling Processes in Working Memory: Instructions Control the Order of Memory Search and Mental Arithmetic , 1997, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[36]  Alexander I. Rudnicky,et al.  Models for evaluating interaction protocols in speech recognition , 1991, CHI.

[37]  Peter Johnson,et al.  Human computer interaction: Psychology, task analysis, and software engineering , 1992 .

[38]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[39]  I.,et al.  Fitts' Law as a Research and Design Tool in Human-Computer Interaction , 1992, Hum. Comput. Interact..