Subjective Usability, Mental Workload Assessments and Their Impact on Objective Human Performance

Self-reporting procedures and inspection methods have been largely employed in the fields of interaction and web-design for assessing the usability of interfaces. However, there seems to be a propensity to ignore features related to end-users or the context of application during the usability assessment procedure. This research proposes the adoption of the construct of mental workload as an additional aid to inform interaction and web-design. A user-study has been performed in the context of human-web interaction. The main objective was to explore the relationship between the perception of usability of the interfaces of three popular web-sites and the mental workload imposed on end-users by a set of typical tasks executed over them. Usability scores computed employing the System Usability Scale were compared and related to the mental workload scores obtained employing the NASA Task Load Index and the Workload Profile self-reporting assessment procedures. Findings advise that perception of usability and subjective assessment of mental workload are two independent, not fully overlapping constructs. They measure two different aspects of the human-system interaction. This distinction enabled the demonstration of how these two constructs cab be jointly employed to better explain objective performance of end-users, a dimension of user experience, and informing interaction and web-design.

[1]  Luca Longo.  Designing Medical Interactive Systems Via Assessment of Human Mental Workload , 2015, 2015 IEEE 28th International Symposium on Computer-Based Medical Systems.

[2]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[3]  Jacek Gwizdka,et al.  Assessing Cognitive Load on Web Search Tasks , 2009, ArXiv.

[4]  A. H. Roscoe,et al.  A Subjective Rating Scale for Assessing Pilot Workload in Flight: A decade of Practical Use , 1990 .

[5]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[6]  Brian Shackel,et al.  Usability - Context, framework, definition, design and evaluation , 1991, Interact. Comput..

[7]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[8]  George E. Cooper,et al.  The use of pilot rating in the evaluation of aircraft handling qualities , 1969 .

[9]  Mounia Lalmas,et al.  Models of user engagement , 2012, UMAP.

[10]  Jacek Gwizdka,et al.  Distribution of cognitive load in Web search , 2010, J. Assoc. Inf. Sci. Technol..

[11]  Luca Longo,et al.  Human-Computer Interaction and Human Mental Workload: Assessing Cognitive Engagement in the World Wide Web , 2011, INTERACT.

[12]  Luca Longo,et al.  A novel methodology for evaluating user interfaces in health care , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[13]  P. Tsang,et al.  Diagnosticity and multidimensional subjective workload ratings. , 1996, Ergonomics.

[14]  Barry H. Kantowitz,et al.  Mental Workload , 2020, Encyclopedia of Behavioral Medicine.

[15]  John Annett,et al.  Subjective rating scales in ergonomics: a reply , 2002 .

[16]  Pamela S. Tsang,et al.  Mental Workload and Situation Awareness , 2006 .

[17]  W. Cleveland.  Robust Locally Weighted Regression and Smoothing Scatterplots , 1979 .

[18]  Waldemar Karwowski,et al.  Mental Workload Measurement , 2006 .

[19]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[20]  Luca Longo,et al.  A Computational Analysis of Cognitive Effort , 2010, ACIIDS.

[21]  D. Novis.  In Reply. , 2016, Archives of pathology & laboratory medicine.

[22]  Kurt Hornik,et al.  Support Vector Machines in R , 2006 .

[23]  Leif Azzopardi,et al.  The Impact of Query Interface Design on Stress, Workload and Performance , 2015, ECIR.

[24]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[25]  Luca Longo,et al.  On the Relationship between Perception of Usability and Subjective Mental Workload of Web Interfaces , 2015, 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT).

[26]  Fabio Rusconi,et al.  The Importance of Human Mental Workload in Web Design , 2012, WEBIST.

[27]  Dick de Waard,et al.  The measurement of drivers' mental workload , 1996 .

[28]  G. Salvendy,et al.  Review and reappraisal of modelling and predicting mental workload in single- and multi-task environments , 2000 .

[29]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[30]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[31]  Michael J. Albers.  Tapping as a measure of cognitive load and website usability , 2011, SIGDOC '11.

[32]  Miles Macleod,et al.  Usability in Context: Improving Quality of Use , 1994 .

[33]  Kristin P. Bennett,et al.  Support vector machines: hype or hallelujah? , 2000, SKDD.

[34]  Luca Longo,et al.  Modeling Mental Workload Via Rule-Based Expert System: A Comparison with NASA-TLX and Workload Profile , 2016, AIAI.

[35]  P. D Harper,et al.  Improving user satisfaction: the questionnaire for user interaction satisfaction version 5 , 1993 .

[36]  Luca Longo,et al.  Formalising human mental workload as non-monotonic concept for adaptive and personalised web-design , 2012, UMAP.

[37]  Silvia Heinz,et al.  Cognitive Load in eCommerce Applications - Measurement and Effects on User Satisfaction , 2009, Adv. Hum. Comput. Interact..

[38]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[39]  C. Wickens,et al.  Workload transition: Implications for individual and team performance. , 1993 .

[40]  Gerhard Fischer,et al.  User Modeling in Human–Computer Interaction , 2001, User Modeling and User-Adapted Interaction.

[41]  Thomas E. Nygren,et al.  The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload , 1988 .

[42]  Luca Longo,et al.  Formalising Human Mental Workload as a Defeasible Computational Concept , 2014 .

[43]  Alex Endert,et al.  Beyond Usability and Performance: A Review of User Experience-focused Evaluations in Visualization , 2016, BELIV '16.

[44]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[45]  Chris Baber,et al.  Subjective evaluation of usability , 2002, Ergonomics.

[46]  Saturnino Luz,et al.  Human Mental Workload: Models and Applications , 2017, Communications in Computer and Information Science.

[47]  Vicente Moret-Bonillo,et al.  A Context-of-Use Taxonomy for Usability Studies , 2010, Int. J. Hum. Comput. Interact..

[48]  Christopher D. Wickens,et al.  Multiple Resources and Mental Workload , 2008, Hum. Factors.

[49]  Luca Longo,et al.  A defeasible reasoning framework for human mental workload representation and assessment , 2015, Behav. Inf. Technol..

[50]  C. Wickens.  Engineering psychology and human performance, 2nd ed. , 1992 .

[51]  J. Annett.  Subjective rating scales: science or art? , 2002, Ergonomics.

[52]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[53]  B. Cain.  A Review of the Mental Workload Literature , 2007 .

[54]  Luca Longo,et al.  Mental Workload in Medicine: Foundations, Applications, Open Problems, Challenges and Future Perspectives , 2016, 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS).

[55]  Haibin Zhu,et al.  Restrain mental workload with roles in HCI , 2009, 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH).

[56]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[57]  F. Thomas Eggemeier,et al.  Workload assessment methodology. , 1986 .

[58]  Thomas S. Tullis,et al.  A Comparison of Questionnaires for Assessing Website Usability , 2004 .

[59]  James C. Schueren,et al.  Using the Subjective Workload Dominance (SWORD) Technique for Projective Workload Assessment , 1991 .