Using physiological sensors to detect levels of user frustration induced by system delays

In mobile computing, varying access to resources makes it difficult for developers to ensure that satisfactory system response times will be maintained at all times. Wearable physiological sensors offer a way to dynamically detect user frustration in response to increased system delays. However, most prior efforts have focused on binary classifiers designed to detect the presence or absence of a task-specific stimulus. In this paper, we make two contributions. Our first contribution is in identifying the use of variable length system response delays, a universal and task-independent feature of computing, as a stimulus for driving different levels of frustration. By doing so, we are able to make our second and primary contribution, which is the development of models that predict multiple levels of user frustration from psycho-physiological responses caused by system response delays. We investigate how incorporating different sensor features, application settings, and timing constraints impact the performance of our models. We demonstrate that our models of physiological responses can be used to classify five levels of frustration in near real-time with over 80% accuracy, which is comparable to the accuracy of binary classifiers.

[1]  Jodi Forlizzi,et al.  Psycho-physiological measures for assessing cognitive load , 2010, UbiComp.

[2]  Yuan Qi,et al.  The Bayes Point Machine for computer-user frustration detection via pressuremouse , 2001, PUI '01.

[3]  Mahadev Satyanarayanan,et al.  vTube: efficient streaming of virtual appliances over last-mile networks , 2013, SoCC.

[4]  Francoise Beaufays,et al.  Google Search by Voice: A Case Study , 2010 .

[5]  Daniel P. Siewiorek,et al.  Emotion Recognition Modulating the Behavior of Intelligent Systems , 2013, 2013 IEEE International Symposium on Multimedia.

[6]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[7]  Pat Hanrahan,et al.  Measuring the task-evoked pupillary response with a remote eye tracker , 2008, ETRA.

[8]  Ashwin Belle,et al.  Frustration Detection with Electrocardiograph Signal Using Wavelet Transform , 2010, 2010 International Conference on Biosciences.

[9]  Mitchell Grossberg,et al.  An Experiment on Problem Solving with Delayed Computer Responses , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[10]  A. Mehrabian Basic dimensions for a general psychological theory : implications for personality, social, environmental, and developmental studies , 1980 .

[11]  I. Lazar,et al.  The state of the Internet , 2000 .

[12]  Francoise Beaufays,et al.  “Your Word is my Command”: Google Search by Voice: A Case Study , 2010 .

[13]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[14]  Ashish Kapoor,et al.  Automatic prediction of frustration , 2007, Int. J. Hum. Comput. Stud..

[15]  W. Kuhmann Experimental investigation of stress-inducing properties of system response times. , 1989, Ergonomics.

[16]  T. W. Butler Computer response time and user performance. , 1983, CHI '83.

[17]  Andrea Kleinsmith,et al.  Affective Body Expression Perception and Recognition: A Survey , 2013, IEEE Transactions on Affective Computing.

[18]  Yuan Gao,et al.  What Does Touch Tell Us about Emotions in Touchscreen-Based Gameplay? , 2012, TCHI.

[19]  Jan L. Guynes Impact of system response time on state anxiety , 1988, CACM.

[20]  Melvin L. Snyder,et al.  Time Urgency and the Type A Coronary‐Prone Behavior Pattern1 , 1974 .

[21]  Amy Neustein Advances in Speech Recognition: Mobile Environments, Call Centers and Clinics , 2010 .

[22]  Gary L. Dannenbring The effect of computer response time on user performance and satisfaction: A preliminary investigation , 1983 .

[23]  John A. Hoxmeier,et al.  System Response Time and User Satisfaction: An Experimental Study of Browser-based Applications , 2000 .

[24]  Fiona Fui-Hoon Nah,et al.  A study on tolerable waiting time: how long are Web users willing to wait? , 2004, AMCIS.

[25]  Jonathan Klein,et al.  Frustrating the user on purpose: a step toward building an affective computer , 2002, Interact. Comput..

[26]  Alexander Gruenstein,et al.  Accurate and compact large vocabulary speech recognition on mobile devices , 2013, INTERSPEECH.

[27]  John Zahorjan,et al.  The challenges of mobile computing , 1994, Computer.

[28]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Michael C. Pyryt Human cognitive abilities: A survey of factor analytic studies , 1998 .

[30]  Winslow Burleson,et al.  Affective agents: Sustaining motivation to learn through failure and state of "stuck" , 2004 .

[31]  Mahadev Satyanarayanan,et al.  Fundamental challenges in mobile computing , 1996, PODC '96.

[32]  L. F. Barrett,et al.  Is there consistency and specificity of autonomic changes during emotional episodes? Guidance from the Conceptual Act Theory and psychophysiology , 2014, Biological Psychology.

[33]  John H. L. Hansen,et al.  Analysis and detection of cognitive load and frustration in drivers' speech , 2010, INTERSPEECH.

[34]  Avi Rushinek,et al.  What makes users happy? , 1986, CACM.

[35]  Xiaodong Cui,et al.  Towards High Performance LVCSR in Speech-to-Speech Translation System on Smart Phones , 2011, INTERSPEECH.

[36]  Paramvir Bahl,et al.  The Case for VM-Based Cloudlets in Mobile Computing , 2009, IEEE Pervasive Computing.

[37]  Thomas E. Kurtz,et al.  The Dartmouth Time-Sharing Computing System. Final Report. , 1967 .

[38]  Scotty D. Craig,et al.  Integrating Affect Sensors in an Intelligent Tutoring System , 2004 .

[39]  Michelle Karg,et al.  Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation , 2013, IEEE Transactions on Affective Computing.

[40]  Mahadev Satyanarayanan,et al.  Quantifying interactive user experience on thin clients , 2006, Computer.

[41]  Ryan Shaun Joazeiro de Baker,et al.  Coarse-grained detection of student frustration in an introductory programming course , 2009, ICER '09.

[42]  Bryan Reimer,et al.  An on-road assessment of the impact of cognitive workload on physiological arousal in young adult drivers , 2009, AutomotiveUI.

[43]  Bharat K. Bhargava,et al.  A Survey of Computation Offloading for Mobile Systems , 2012, Mobile Networks and Applications.

[44]  K. McGrew CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research , 2009 .

[45]  R. Lawson Frustration : the development of a scientific concept , 1965 .