Understanding the Relationship between Frustration and the Severity of Usability Problems: What can Psychophysiological Data (Not) Tell Us?

Frustration is used as a criterion for identifying usability problems (UPs) and for rating their severity in a few of the existing severity scales, but it is not operationalized. No research has systematically examined how frustration varies with the severity of UPs. We aimed to address these issues with a hybrid approach, using Self-Assessment Manikin, comments elicited with Cued-Recall Debrief, galvanic skin responses (GSR) and gaze data. Two empirical studies involving a search task with a website known to have UPs were conducted to substantiate findings and improve on the methodological framework, which could facilitate usability evaluation practice. Results showed no correlation between GSR peaks and severity ratings, but GSR peaks were correlated with frustration scores -- a metric we developed. The Peak-End rule was partially verified. The problematic evaluator effect was the limitation as it confounded the severity ratings of UPs. Future work is aimed to control this effect and to develop a multifaceted severity scale.

[1]  Paul van Schaik,et al.  Attitudes towards user experience (UX) measurement , 2014, Int. J. Hum. Comput. Stud..

[2]  Robin Jeffries,et al.  Usability problem reports: helping evaluators communicate effectively with developers , 1994 .

[3]  G. Wolford,et al.  Evaluations of pleasurable experiences: The peak-end rule , 2008, Psychonomic bulletin & review.

[4]  R. Lazarus.  Progress on a cognitive-motivational-relational theory of emotion. , 1991, The American psychologist.

[5]  L. R. Goldberg.  THE DEVELOPMENT OF MARKERS FOR THE BIG-FIVE FACTOR STRUCTURE , 1992 .

[6]  Ben Shneiderman,et al.  A model for computer frustration: the role of instrumental and dispositional factors on incident, session, and post-session frustration and mood , 2006, Comput. Hum. Behav..

[7]  Lewis R. Goldberg,et al.  Big-Five Factor Markers , 2011 .

[8]  J. McLennan,et al.  Studying Complex Decision Making in Natural Settings: Using a Head-Mounted Video Camera to Study Competitive Orienteering , 1994, Perceptual and motor skills.

[9]  Gavriel Salvendy,et al.  Handbook of Human Factors and Ergonomics , 2005 .

[10]  A. Amsel,et al.  Frustration theory: Subject index , 1992 .

[11]  Morten Hertzum,et al.  What you get is what you see: revisiting the evaluator effect in usability tests , 2014, Behav. Inf. Technol..

[12]  S. Fairclough,et al.  Prediction of subjective states from psychophysiology: A multivariate approach , 2006, Biological Psychology.

[13]  Raul Fernandez,et al.  Stochastic Modeling of Physiological Signals with Hidden Markov Models: A Step Toward Frustration Detection in Human-Computer Interfaces , 1997 .

[14]  Christian Martyn Jones,et al.  Biometric valence and arousal recognition , 2007, OZCHI '07.

[15]  Rosalind W. Picard.  Affective Computing , 1997 .

[16]  J. Andreassi.  Psychophysiology: Human Behavior and Physiological Response , 1980 .

[17]  Steven M. Belz,et al.  The user action framework: a reliable foundation for usability engineering support tools , 2001, Int. J. Hum. Comput. Stud..

[18]  Morten Hertzum,et al.  The evaluator effect in usability tests , 1998, CHI Conference Summary.

[19]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[20]  Tao Lin,et al.  Do physiological data relate to traditional usability indexes? , 2005, OZCHI.

[21]  Jonathan Klein,et al.  This computer responds to user frustration: Theory, design, and results , 2002, Interact. Comput..

[22]  Lorraine Johnston,et al.  Evaluation using cued-recall debrief to elicit information about a user's affective experiences , 2005, OZCHI.

[23]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[24]  Rosalind W. Picard.  Affective computing: (526112012-054) , 1997 .

[25]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[26]  R. Lazarus.  Emotion and Adaptation , 1991 .

[27]  Claude J. Elie,et al.  Remote Usability Evaluation: Overview and Case Studies , 2002, Int. J. Hum. Comput. Interact..

[28]  R. Mandryk.  Modeling user emotion in interactive play environments: a fuzzy physiological approach , 2005 .

[29]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[30]  D. Kahneman,et al.  Duration neglect in retrospective evaluations of affective episodes. , 1993, Journal of personality and social psychology.

[31]  Ben Shneiderman,et al.  Severity and impact of computer user frustration: A comparison of student and workplace users , 2006, Interact. Comput..

[32]  D. Kahneman,et al.  When More Pain Is Preferred to Less: Adding a Better End , 1993 .

[33]  Morten Hertzum,et al.  Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design , 2006, Int. J. Hum. Comput. Interact..

[34]  James R. Lewis.  Introduction: Current Issues in Usability Evaluation , 2001, Int. J. Hum. Comput. Interact..

[35]  J. Sauro.  The relationship between problem frequency and problem severity in usability evaluations , 2014 .

[36]  D. Kahneman.  Thinking, Fast and Slow , 2011 .

[37]  Rebecca A. Grier,et al.  Fundamental dimensions of subjective state in performance settings: task engagement, distress, and worry. , 2002, Emotion.

[38]  David C. Rubin,et al.  Autobiographical Memory , 2019, Encyclopedia of Autism Spectrum Disorders.

[39]  Martina Manhartsberger,et al.  Eye tracking in usability research: What users really see , 2005, Usability Symposium.

[40]  J. Lagopoulos.  Electrodermal activity , 2007, Acta Neuropsychiatrica.

[41]  P. Lang.  Behavioral treatment and bio-behavioral assessment: computer applications , 1980 .

[42]  Marc Hassenzahl.  Prioritizing usability problems: Data-driven and judgement-driven severity estimates , 2000, Behav. Inf. Technol..

[43]  Paul Dourish,et al.  How emotion is made and measured , 2007, Int. J. Hum. Comput. Stud..

[44]  D. Kahneman,et al.  Memories of yesterday's emotions: does the valence of experience affect the memory-experience gap? , 2009, Emotion.

[45]  Klaus Kaasgaard,et al.  Comparative usability evaluation , 2004, Behav. Inf. Technol..

[46]  A. Amsel.  Frustration Theory: An Analysis of Dispositional Learning and Memory , 1992 .

[47]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[48]  Anders Bruun,et al.  Mind the Gap! Comparing Retrospective and Concurrent Ratings of Emotion in User Experience Evaluation , 2015, INTERACT.

[49]  A. Mehrabian.  Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament , 1996 .

[50]  Angela M. Cirucci,et al.  Usability Testing , 2021, UX Research Methods for Media and Communication Studies.

[51]  W. Boucsein.  Electrodermal activity, 2nd ed. , 2012 .

[52]  Carl Gutwin,et al.  Examining the Peak-End Effects of Subjective Experience , 2015, CHI.

[53]  J. Russell.  A circumplex model of affect. , 1980 .

[54]  Jan Stage,et al.  Supporting problem identification in usability evaluations , 2005, OZCHI.

[55]  J. Russell,et al.  The Measurement of Affect, Mood, and Emotion: A Guide for Health-Behavioral Research , 2013 .

[56]  Robert D. Ward,et al.  Physiological responses to different WEB page designs , 2003, Int. J. Hum. Comput. Stud..

[57]  Kasper Hornbæk,et al.  A Study of the Evaluator Effect in Usability Testing , 2008, Hum. Comput. Interact..

[58]  D. Kahneman.  A perspective on judgment and choice: mapping bounded rationality. , 2003, The American psychologist.

[59]  Regan L. Mandryk,et al.  A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies , 2007, Int. J. Hum. Comput. Stud..

[60]  Sari Kujala,et al.  Emotions, experiences and usability in real-life mobile phone use , 2013, CHI.

[61]  Linden J. Ball,et al.  Eye tracking in HCI and usability research. , 2006 .

[62]  R. Lawson.  Frustration : the development of a scientific concept , 1965 .

[63]  Kristina Höök,et al.  CHI '12 Extended Abstracts on Human Factors in Computing Systems , 2012, CHI 2012.

[64]  Yuan Gao,et al.  What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay? , 2012, TCHI.

[65]  K. J. Vicente,et al.  Spectral Analysis of Sinus Arrhythmia: A Measure of Mental Effort , 1987, Human factors.

[66]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[67]  Joseph S. Dumas,et al.  Comparative usability evaluation (CUE-4) , 2008, Behav. Inf. Technol..

[68]  P. Foglia,et al.  Relating GSR Signals to traditional Usability Metrics: Case Study with an anthropomorphic Web Assistant , 2008, 2008 IEEE Instrumentation and Measurement Technology Conference.

[69]  Virpi Roto,et al.  USER EXPERIENCE WHITE PAPER Bringing clarity to the concept of user experience , 2011 .

[70]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[71]  Malin Forne,et al.  Physiology as a Tool for UX and Usability Testing – A comparative study of pupil size and other physiological measures. , 2012 .

[72]  Marc Hassenzahl,et al.  From mental effort to perceived usability: transforming experiences into summary assessments , 2004, CHI EA '04.

[73]  Christopher D. B. Burt,et al.  A test of the peak-end rule with extended autobiographical events , 2008, Memory & cognition.

[74]  Fang Chen,et al.  Galvanic skin response (GSR) as an index of cognitive load , 2007, CHI Extended Abstracts.

[75]  Wendy D Roth,et al.  The Rashomon Effect , 2002 .

[76]  Bonnie E. John,et al.  Tracking the effectiveness of usability evaluation methods , 1997, Behav. Inf. Technol..

[77]  Lei Zhou,et al.  Using Physiological Measures to Evaluate User Experience of Mobile Applications , 2014, HCI.

[78]  D. Kahneman,et al.  Patients' memories of painful medical treatments: real-time and retrospective evaluations of two minimally invasive procedures , 1996, Pain.

[79]  Abram Amsel.  Frustration theory: Contents , 1992 .

[80]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[81]  D. Cicchetti.  Emotion and Adaptation , 1993 .