Affective state detection via facial expression analysis within a human–computer interaction context

The advancement in technology indicates that there is an opportunity to enhance human–computer interaction by way of affective state recognition. Affective state recognition is typically based on passive stimuli such as watching video clips, which does not reflect genuine interaction. This paper presents a study on affective state recognition using active stimuli, i.e. facial expressions of users when they attempt computerised tasks, particularly across typical usage of computer systems. A data collection experiment is presented for acquiring data from normal users whilst they interact with software, attempting to complete a set of predefined tasks. In addition, a hierarchical machine learning approach is presented for facial expression-based affective state recognition, which employs an Euclidean distance-based feature representation, conjointly with a customised encoding for users’ self-reported affective states. Consequently, the aim is to find the potential relationship between the facial expressions, as defined by Paul Ekman, and the self-reported emotional states specified by users using Russells Circumplex model, in relation to the actual feelings and affective states. The main findings of this study suggest that facial expressions cannot precisely reveal the actual feelings of users whilst interacting with common computerised tasks. Moreover, during active interaction tasks more variation occurs within the facial expressions of participants than occurs within passive interaction.

[1]  Yvonne Rogers,et al.  Being Human: Human-Computer Interaction in the Year 2020 , 2019 .

[2]  David W. Aha,et al.  Generalizing from Case studies: A Case Study , 1992, ML.

[3]  Stefanos Zafeiriou,et al.  Incremental Face Alignment in the Wild , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Nicu Sebe,et al.  Communication and Automatic Interpretation of Affect from Facial Expressions , 2010 .

[5]  Hui Wang,et al.  Sensing Affective States Using Facial Expression Analysis , 2016, UCAmI.

[6]  Christian D. Schunn,et al.  Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction , 2002, Proc. IEEE.

[7]  Carlos D. Castillo,et al.  Facial Action Coding , 2009, Encyclopedia of Biometrics.

[8]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[9]  J. Coull.  Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology , 1998, Progress in Neurobiology.

[10]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[11]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[12]  Hyunseung Choo,et al.  Evaluating real-life performance of the state-of-the-art in facial expression recognition using a novel YouTube-based datasets , 2016, Multimedia Tools and Applications.

[13]  T. Robbins.  Arousal systems and attentional processes , 1997, Biological Psychology.

[14]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[15]  A. M. Martinez,et al.  Deciphering the face , 2011, CVPR 2011 WORKSHOPS.

[16]  Nicu Sebe,et al.  Exploiting facial expressions for affective video summarisation , 2009, CIVR '09.

[17]  Yvonne Rogers,et al.  Being Human: HCI in the Year 2020. , 2008 .

[18]  Mary Czerwinski,et al.  Under pressure: sensing stress of computer users , 2014, CHI.

[19]  Takehisa Yairi,et al.  Facial Expression Recognition and Analysis: A Comparison Study of Feature Descriptors , 2015, IPSJ Trans. Comput. Vis. Appl..

[20]  D. Lundqvist,et al.  Karolinska Directed Emotional Faces , 2015 .

[21]  Milad Alemzadeh,et al.  Human-Computer Interaction: Overview on State of the Art , 2008 .

[22]  Edilson de Aguiar,et al.  Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order , 2017, Pattern Recognit..

[23]  Thomas Beauvisage,et al.  Computer usage in daily life , 2009, CHI.

[24]  Rosalind W. Picard.  Affective computing: challenges , 2003, Int. J. Hum. Comput. Stud..

[25]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  J. Russell.  A circumplex model of affect. , 1980 .