Towards Multimodal Error Management: Experimental Evaluation of User Strategies in Event of Faulty Application Behavior in Automotive Environments

In this work, we present the results of a study analyzing the reactions of subjects on simulated errors of a dedicated in-car interface for controlling infotainment and communication services. The test persons could operate the system, using different input modalities, such as natural or command speech as well as head and hand gestures, or classical tactile paradigms. In various situational contexts, we scrutinized the interaction patterns the test participants applied to overcome different operation tasks. Moreover, we evaluated individual user behavior concerning modality transitions and individual fallback strategies in case of system errors. Two different error types (Hidden System Errors and Apparent System Errors) were provoked. As a result, we found out that initially, i.e. with the system working properly, most users prefer tactile or speech interaction. In case of Hidden System Errors, mostly changes from speech to tactile interaction and vice versa occurred. Concerning Apparent System Errors, 87% of the subjects automatically interrupted or cancelled their input procedure. 73% of all test persons who continued interaction, when the reason for the faulty system behavior was gone, strictly kept the selected modality. Regarding the given input vocabulary, none of the subjects selected head or hand gesture input as the leading fallback modality.

[1]  D. Meister.  The nature of human error , 1989, IEEE Global Telecommunications Conference, 1989, and Exhibition. 'Communications Technology for the 1990s and Beyond.

[2]  M. Lang,et al.  Gestenbasierte Interaktion mit Geräten im Automobil - Gesture-Based Control of In-Car Devices , 2002 .

[3]  G. McGlaun,et al.  Ein neuer Systemansatz für die Integration multimodalen Inputs durch Late Semantic Fusion - A new Approach for Integrating Multimodal Input via Late Semantic Fusion , 2002 .

[4]  B. Schuller,et al.  Towards intuitive speech interaction by the integration of emotional aspects , 2002, IEEE International Conference on Systems, Man and Cybernetics.

[5]  L. Festinger.  A Theory of Cognitive Dissonance , 1957 .

[6]  Sharon L. Oviatt,et al.  Taming recognition errors with a multimodal interface , 2000, CACM.

[7]  Björn W. Schuller,et al.  A new technique for adjusting distraction moments in multitasking non-field usability tests , 2002, CHI Extended Abstracts.

[8]  M. Lang,et al.  Development of a Generic Multimodal Framework for Handling Error Patterns during Human-Machine Interaction , 2004 .

[9]  Jens Rasmussen,et al.  Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[10]  J. Reason,et al.  Modelling the basic error tendencies of human operators , 1988 .

[11]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[12]  Sharon L. Oviatt,et al.  Error resolution during multimodal human-computer interaction , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[13]  Sharon L. Oviatt,et al.  Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions , 2000, Hum. Comput. Interact..

[14]  M. Lang,et al.  Bedienkonzept zur Gestenbasierten Interaktion mit Geräten im Automobil - Gesture-Based Control Concept for In-Car Devices , 2002 .

[15]  Sharon L. Oviatt,et al.  Mutual disambiguation of recognition errors in a multimodel architecture , 1999, CHI '99.

[16]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[17]  Peter Morguet,et al.  Comparison of approaches to continuous hand gesture recognition for a visual dialog system , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[18]  Gerhard Rigoll,et al.  Gesture Components for Natural Interaction with In-Car Devices , 2003, Gesture Workshop.

[19]  Björn Schuller,et al.  Towards automation of usability studies , 2002, IEEE International Conference on Systems, Man and Cybernetics.

[20]  Björn W. Schuller,et al.  Experimental evaluation of user errors at the skill-based level in an automative environment , 2002, CHI Extended Abstracts.

[21]  Lisa J. Servon.  Bridging the Digital Divide , 2002 .