Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing

There is an agreement that perceived usability is important beyond actual effectiveness of software systems. Perceived usability is often obtained by self-reports provided after system use. Aiming to improve summative usability testing, we propose a methodology to enhance in-depth testing of users' performance and perceived usability at the task level. The metacognitive research approach allows detailed analysis of cognitive processes. Adapting its methodologies, we propose the Metacognitive Usability Profile (MUP) which includes a comprehensive set of measures based on collecting confidence in the success of each particular task and triangulating it with objective measures. We demonstrate using the MUP by comparing two versions of a project management system. Based on a task analysis we allocated tasks that differ between the versions and let participants (N?=?100) use both versions. Although no difference was found between the versions in system-level perceived usability, the detailed task-level analysis exposed many differences. In particular, overconfidence was associated with low performance, which suggests that user interfaces better avoid illusions of knowing. Overall, the study demonstrates how the MUP exposes challenges users face. This, in turn, allows choosing the better task implementation among the examined options and to focus attempts for usability improvement. Metacognitive approach allows a detailed analysis of perceived usability and performance.We propose the Metacognitive Usability Profile (MUP) for a detailed usability test.Test output is a scorecard that allows task-level comparisons of user interfaces.We demonstrate MUP use by testing two versions of a project management system.Conclusion: MUP exposed differences otherwise concealed.

[1]  Jürgen Sauer,et al.  The influence of user expertise and prototype fidelity in usability tests. , 2010, Applied ergonomics.

[2]  Tom Tullis,et al.  Measuring the User Experience, Second Edition: Collecting, Analyzing, and Presenting Usability Metrics , 2013 .

[3]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[4]  Rakefet Ackerman,et al.  Taking reading comprehension exams on screen or on paper? A metacognitive analysis of learning texts under time pressure , 2012, Comput. Hum. Behav..

[5]  Janne Jul Jensen,et al.  A case study of three software projects: can software developers anticipate the usability problems in their software? , 2008, Behav. Inf. Technol..

[6]  James R. Lewis,et al.  Usability: Lessons Learned … and Yet to Be Learned , 2014, Int. J. Hum. Comput. Interact..

[7]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[8]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[9]  J. Metcalfe,et al.  People's hypercorrection of high-confidence errors: did they know it all along? , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[10]  D. S. Lindsay,et al.  Remembering Mistaken for Knowing: Ease of Retrieval as a Basis for Confidence in Answers to General Knowledge Questions , 1993 .

[11]  Marc Hassenzahl,et al.  From mental effort to perceived usability: transforming experiences into summary assessments , 2004, CHI EA '04.

[12]  Rakefet Ackerman,et al.  Metacognitive regulation of text learning: on screen versus on paper. , 2011, Journal of experimental psychology. Applied.

[13]  Avraham Shtub,et al.  Simulation Based Training (SBT) - the Next Generation of Project Management Training , 2013 .

[14]  Asher Koriat,et al.  The intricate relationships between monitoring and control in metacognition: lessons for the cause-and-effect relation between subjective experience and behavior. , 2006, Journal of experimental psychology. General.

[15]  Erin Dorris Cassidy,et al.  Student Searching with EBSCO Discovery: A Usability Study , 2014 .

[16]  Colleen M. Kelley,et al.  Solve the problem first: constructive solution strategies can influence the accuracy of retrospective confidence judgments. , 2010, Journal of experimental psychology. Learning, memory, and cognition.

[17]  Rakefet Ackerman,et al.  The effects of goal-driven and data-driven regulation on metacognitive monitoring during learning: a developmental perspective. , 2014, Journal of experimental psychology. General.

[18]  Gitte Lindgaard,et al.  Usability testing: what have we overlooked? , 2007, CHI.

[19]  Anna Sagana,et al.  Assessing nonchoosers' eyewitness identification accuracy from photographic showups by using confidence and response times. , 2012, Law and human behavior.

[20]  James R. Lewis Introduction to the Special Issue on Usability and User Experience: Methodological Evolution , 2015, Int. J. Hum. Comput. Interact..

[21]  Matt Jones,et al.  ONTRACK: Dynamically adapting music playback to support navigation , 2008, Personal and Ubiquitous Computing.

[22]  R. Bjork Memory and metamemory considerations in the training of human beings. , 1994 .

[23]  Robert A. Bjork,et al.  When disfluency is—and is not—a desirable difficulty: The influence of typeface clarity on metacognitive judgments and memory , 2013, Memory & cognition.

[24]  Jeff Sauro,et al.  Correlations among prototypical usability metrics: evidence for the construct of usability , 2009, CHI.

[25]  R. Bjork,et al.  Self-regulated learning: beliefs, techniques, and illusions. , 2013, Annual review of psychology.

[26]  Kraig Finstad,et al.  The Usability Metric for User Experience , 2010, Interact. Comput..

[27]  Rakefet Ackerman,et al.  The persistence of the fluency–confidence association in problem solving , 2012, Psychonomic bulletin & review.

[28]  Tingting Zhao,et al.  Dual Verbal Elicitation: The Complementary Use of Concurrent and Retrospective Reporting Within a Usability Test , 2013, Int. J. Hum. Comput. Interact..

[29]  Philip Kortum,et al.  The Relationship Between Task-level and Test-level System Usability Scale Scores , 2014 .

[30]  James R. Lewis,et al.  Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ , 1991, SGCH.

[31]  Morten Hertzum,et al.  Thinking Aloud in the Presence of Interruptions and Time Constraints , 2013, Int. J. Hum. Comput. Interact..

[32]  David J. Therriault,et al.  Accuracy of metacognitive monitoring affects learning of texts. , 2003 .

[33]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[34]  James R. Lewis,et al.  Measuring Perceived Usability: The SUS, UMUX-LITE, and AltUsability , 2015, Int. J. Hum. Comput. Interact..

[35]  Henrik Olsson,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000 .

[36]  John Dunlosky,et al.  Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention , 2012 .

[37]  Gilbert Cockton,et al.  Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation , 2001, BCS HCI/IHM.

[38]  Joseph S. Dumas,et al.  Comparison of three one-question, post-task usability questionnaires , 2009, CHI.

[39]  James R. Lewis,et al.  Development and Psychometric Evaluation of the Emotional Metric Outcomes (EMO) Questionnaire , 2014, Int. J. Hum. Comput. Interact..

[40]  Morten Hertzum,et al.  Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? , 2009, Behav. Inf. Technol..

[41]  T. O. Nelson,et al.  A comparison of current measures of the accuracy of feeling-of-knowing predictions , 1984 .

[42]  Avi Parush,et al.  Simulation‐based Learning in Engineering Education: Performance and Transfer in Learning Project Management , 2006 .

[43]  J. Metcalfe,et al.  Evidence that judgments of learning are causally related to study choice , 2008, Psychonomic bulletin & review.

[44]  T. O. Nelson,et al.  A comparison of current measures of the accuracy of feeling-of-knowing predictions. , 1984, Psychological bulletin.

[45]  P. Juslin,et al.  Naive empiricism and dogmatism in confidence research: a critical examination of the hard-easy effect. , 2000, Psychological review.

[46]  J. Metcalfe Cognitive Optimism: Self-Deception or Memory-Based Processing Heuristics? , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[47]  J. Metcalfe,et al.  Errors committed with high confidence are hypercorrected. , 2001, Journal of experimental psychology. Learning, memory, and cognition.

[48]  Rakefet Ackerman,et al.  Response latency as a predictor of the accuracy of children's reports. , 2011, Journal of experimental psychology. Applied.

[49]  Stefano Federici,et al.  Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience , 2015, Int. J. Hum. Comput. Interact..

[50]  Rakefet Ackerman,et al.  The diminishing criterion model for metacognitive regulation of time investment. , 2014, Journal of experimental psychology. General.

[51]  Cathleen Wharton,et al.  Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces , 1990, CHI '90.

[52]  J. Suls,et al.  Flawed Self-Assessment , 2004, Psychological science in the public interest : a journal of the American Psychological Society.

[53]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[54]  James R. Lewis Introduction to the Special Issue on Usability and User Experience: Psychometrics , 2015, Int. J. Hum. Comput. Interact..

[55]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[56]  Morten Hertzum,et al.  Images of Usability , 2010, Int. J. Hum. Comput. Interact..

[57]  John J. Bosley Creating a Short Usability Metric for User Experience (UMUX) Scale , 2013, Interact. Comput..

[58]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[59]  Jennifer Wiley,et al.  Test expectancy affects metacomprehension accuracy. , 2011, The British journal of educational psychology.

[60]  Rakefet Ackerman,et al.  Overcoming screen inferiority in learning and calibration , 2014, Comput. Hum. Behav..

[61]  Philippe A. Palanque,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 2014, International Conference on Human Factors in Computing Systems.

[62]  Caren M Rotello,et al.  Sources of bias in the Goodman-Kruskal gamma coefficient measure of association: implications for studies of metacognitive processes. , 2009, Journal of experimental psychology. Learning, memory, and cognition.

[63]  John T. E. Richardson,et al.  Eta Squared and Partial Eta Squared as Measures of Effect Size in Educational Research. , 2011 .