Subjective Measurement of Workload Related to a Multimodal Interaction Task: NASA-TLX vs. Workload Profile

This paper addresses workload evaluation in the framework of a multimodal application. Two multidimensional subjective workload rating instruments are compared. The goal is to analyze the diagnostics obtained on four implementations of an applicative task. In addition, an Automatic Speech Recognition (ASR) error was introduced in one of the two trials. Eighty subjects participated in the experiment. Half of them rated their subjective workload with NASA-TLX and the other half rated it with Workload Profile (WP) enriched with two stress-related scales. Discriminant and variance analyses revealed a better sensitivity with WP. The results obtained with this instrument led to hypotheses on the cognitive activities of the subjects during interaction. Furthermore, WP permitted us to classify two strategies offered for error recovery. We conclude that WP is more informative for the task tested. WP seems to be a better diagnostic instrument in multimodal system conception.

[1]  P. Tsang,et al.  Diagnosticity and multidimensional subjective workload ratings. , 1996, Ergonomics.

[2]  Laurence Nigay,et al.  A platform for output dialogic strategies in natural multimodal dialogue systems , 2007, IUI '07.

[3]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[4]  Christian A. Müller,et al.  Recognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study , 2001, User Modeling.

[5]  Jan Noyes,et al.  Workload and the use of automatic speech recognition: The effects of time and resource demands , 1996, Speech Commun..

[6]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[7]  Jrg Nievergelt,et al.  Sites, modes, and trails: Telling the user of an interactive system where he is, what he can do, and how to get to places (excerpt) , 1987 .

[8]  Sharon L. Oviatt,et al.  When do we interact multimodally?: cognitive load and multimodal communication patterns , 2004, ICMI '04.

[9]  Raja Parasuraman,et al.  Varieties of attention , 1984 .

[10]  Jean-François Rouet,et al.  Mode and modal transfer effects on performance and discourse organization with an information retrieval dialogue system in natural language , 2006, Comput. Hum. Behav..

[11]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[12]  Laurence Nigay,et al.  Natural multimodal dialogue systems: a configurable dialogue and presentation strategies component , 2007, ICMI '07.

[13]  S. Folkman,et al.  Stress, appraisal, and coping , 1974 .

[14]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.