Measuring Subjective Workload: When Is One Scale Better Than Many?

The idea that mental workload should be treated as a multidimensional problem is well recognized. Yet the NASA Task Load Index and Subjective Workload Assessment Technique scales incorporate procedures for combining multidimensional judgments into scalar estimates of overall workload. In view of comparable or superior performance claimed for univariate scales, it is reasonable to ask whether the scaled values are good estimates of overall processing demands and whether workload values derived from multidimensional scales provide better estimates than do judgments made on a univariate scale. This paper examines these issues using data from four independent studies. The results support two conclusions. First, if a measure of the overall demand on human information-processing is required, then a univariate rating is expected to provide a measure that is at least as sensitive to manipulations of task demand as a derived estimate from multivariate data. Second, if a univariate workload rating is not available, a simple unweighted additive model provides an adequate method for combining factor ratings into an estimate of overall workload.

[1]  F. Mosteller,et al.  Remarks on the method of paired comparisons: III. A test of significance for paired comparisons when equal standard deviations and equal correlations are assumed , 1951, Psychometrika.

[2]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[3]  Michael C. Corballis,et al.  Beyond tests of significance: Estimating strength of effects in selected ANOVA designs. , 1969 .

[4]  Michael H. Birnbaum,et al.  The devil rides again: Correlation as an index of fit. , 1973 .

[5]  B. J. Winer,et al.  Statistical Principles in Experimental Design, 2nd Edition. , 1973 .

[6]  D. H. Dodd,et al.  Computational procedures for estimating magnitude of effect for some analysis of variance designs. , 1973 .

[7]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[8]  J. Dwyer.  Analysis of variance and the magnitude of effects: A general approach. , 1974 .

[9]  F R Brigham,et al.  Some quantitative considerations in questionnaire design and analysis. , 1975, Applied ergonomics.

[10]  John W. Senders.  Axiomatic Models of Workload , 1979 .

[11]  A. Rault,et al.  Measurement of Pilot Workload , 1979 .

[12]  Andries F. Sanders,et al.  Some Remarks on Mental Load , 1979 .

[13]  西里 静彦,et al.  Analysis of categorical data : dual scaling and its applications , 1980 .

[14]  Barry Fowler,et al.  The Aircraft Landing Test: An Information Processing Approach to Pilot Selection , 1981 .

[15]  H. Birx,et al.  The Mismeasure of Man , 1981 .

[16]  W. Wierwille,et al.  A Comparison of Rating Scale, Secondary-Task, Physiological, and Primary-Task Workload Estimation Techniques in a Simulated Flight Task Emphasizing Communications Load , 1983, Human factors.

[17]  Matthias Burisch,et al.  Approaches to personality inventory construction: A comparison of merits. , 1984 .

[18]  D. Gopher,et al.  On the Psychophysics of Workload: Why Bother with Subjective Measures? , 1984 .

[19]  F. Thomas Eggemeier,et al.  Workload Measurement in System Design and Evaluation , 1985 .

[20]  R. Cranley,et al.  Multivariate Analysis—Methods and Applications , 1985 .

[21]  Daniel Gopher,et al.  Workload: An examination of the concept. , 1986 .

[22]  Gary B. Reid,et al.  A Comparative Evaluation of Two Subjective Workload Measures: The Subjective Workload Assessment Technique and the Modified Cooper Harper Scale. , 1986 .

[23]  David W. Biers,et al.  Alternative Approaches to Analyzing SWAT Data , 1987 .

[24]  Pamela S. Tsang,et al.  Absolute Magnitude Estimation and Relative Judgement Approaches to Subjective Workload Assessment , 1987 .

[25]  Henry R. Jex,et al.  Measuring Mental Workload: Problems, Progress, and Promises , 1988 .

[26]  David W. Biers,et al.  An Alternative to Measuring Subjective Workload: Use of SWAT without the Card Sort , 1988 .

[27]  Susan G. Hill,et al.  Workload Assessment of a Remotely Piloted Vehicle (RPV) System , 1988 .

[28]  N. Meshkati,et al.  Toward development of a cohesive model of workload , 1988 .

[29]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[30]  Thomas E. Nygren,et al.  The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload , 1988 .

[31]  Susan G. Hill,et al.  Workload Assessment of a Mobile Air Defense Missile System , 1988 .

[32]  A. O. Dick,et al.  Operator Workload: Comprehensive Review and Evaluation of Operator Workload Methodologies , 1989 .

[33]  M. Appelbaum,et al.  Psychometric methods. , 1989, Annual review of psychology.

[34]  Donald Hameluck,et al.  Instrument Scanning and Subjective Workload with the Peripheral Vision Horizon Display , 1989 .

[35]  Harold R. Booher,et al.  Manprint: An Approach to Systems Integration , 1990 .

[36]  Christopher D. Wickens,et al.  Workload Assessment and Prediction , 1990 .

[37]  Thomas E. Nygren,et al.  Psychometric Properties of Subjective Workload Measurement Techniques: Implications for Their Use in the Assessment of Perceived Mental Workload , 1991 .

[38]  J. C. Byers,et al.  Comparison of Four Subjective Workload Rating Scales , 1992 .

[39]  Lawrence A. Shapiro.  What is Psychophysics? , 1994, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.