Learning deep physiological models of affect

More than 15 years after the early studies in Affective Computing (AC), [1] the problem of detecting and modeling emotions in the context of human-computer interaction (HCI) remains complex and largely unexplored. The detection and modeling of emotion is, primarily, the study and use of artificial intelligence (AI) techniques for the construction of computational models of emotion. The key challenges one faces when attempting to model emotion [2] are inherent in the vague definitions and fuzzy boundaries of emotion, and in the modeling methodology followed. In this context, open research questions are still present in all key components of the modeling process. These include, first, the appropriateness of the modeling tool employed to map emotional manifestations and responses to annotated affective states; second, the processing of signals that express these manifestations (i.e., model input); and third, the way affective annotation (i.e., model output) is handled. This paper touches upon all three key components of an affective model (i.e., input, model, output) and introduces the use of deep learning (DL) [3], [4], [5] methodologies for affective modeling from multiple physiological signals.

[1]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[2]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[3]  Josef Schneeberger,et al.  Interview mit Prof. Dr. Michael M. Richter , 2005, Künstliche Intell..

[4]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[6]  Maria E. Jabon,et al.  Real-time classification of evoked emotions using facial feature tracking and physiological responses , 2008, Int. J. Hum. Comput. Stud..

[7]  E. Vyzas,et al.  Affective Pattern Classification , 2002 .

[8]  Jon D. Morris Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[9]  Derek C. Rose,et al.  Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier] , 2010, IEEE Computational Intelligence Magazine.

[10]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[11]  Pascal Vincent,et al.  Disentangling Factors of Variation for Facial Expression Recognition , 2012, ECCV.

[12]  Hatice Gunes,et al.  Bi-modal emotion recognition from expressive face and body gestures , 2007, J. Netw. Comput. Appl..

[13]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[14]  Samy Bengio,et al.  Inferring document similarity from hyperlinks , 2005, CIKM '05.

[15]  Omar AlZoubi,et al.  Classification of EEG for Affect Recognition: An Adaptive Approach , 2009, Australasian Conference on Artificial Intelligence.

[16]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..

[17]  Geoffrey E. Hinton,et al.  Autoencoders, Minimum Description Length and Helmholtz Free Energy , 1993, NIPS.

[18]  Johannes Wagner,et al.  From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[19]  J. Russell A circumplex model of affect. , 1980 .

[20]  Ashish Kapoor,et al.  Automatic prediction of frustration , 2007, Int. J. Hum. Comput. Stud..

[21]  Eyke Hüllermeier,et al.  Preference Learning , 2005, Künstliche Intell..

[22]  Eyke Hllermeier,et al.  Preference Learning , 2010 .

[23]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[24]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[25]  Pascal Vincent,et al.  Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives , 2012, ArXiv.

[26]  James C. Lester,et al.  Modeling self-efficacy in intelligent tutoring systems: An inductive approach , 2008, User Modeling and User-Adapted Interaction.

[27]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[28]  Javier Hernandez,et al.  Call Center Stress Recognition with Person-Specific Models , 2011, ACII.

[29]  Kristy Elizabeth Boyer,et al.  Predicting Facial Indicators of Confusion with Hidden Markov Models , 2011, ACII.

[30]  Regan L. Mandryk,et al.  A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies , 2007, Int. J. Hum. Comput. Stud..

[31]  Fumio Hara,et al.  Dynamic recognition of basic facial expressions by discrete-time recurrent neural network , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[32]  Craig A. Smith,et al.  From appraisal to emotion: Differences among unpleasant feelings , 1988 .

[33]  Shrikanth S. Narayanan,et al.  Toward detecting emotions in spoken dialogs , 2005, IEEE Transactions on Speech and Audio Processing.

[34]  Mohammed J. Zaki,et al.  Mining features for sequence classification , 1999, KDD '99.

[35]  Yoshua Bengio,et al.  Regularized Auto-Encoders Estimate Local Statistics , 2012, ICLR.

[36]  Kostas Karpouzis,et al.  Detecting human behavior emotional cues in Natural Interaction , 2011, 2011 17th International Conference on Digital Signal Processing (DSP).

[37]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[38]  Constantine Kotropoulos,et al.  Automatic speech classification to five emotional states based on gender information , 2004, 2004 12th European Signal Processing Conference.

[39]  Dimitrios Tzovaras,et al.  Automatic Recognition of Boredom in Video Games Using Novel Biosignal Moment-Based Features , 2011, IEEE Transactions on Affective Computing.

[40]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[41]  Georgios N. Yannakakis,et al.  Towards affective camera control in games , 2010, User Modeling and User-Adapted Interaction.

[42]  Andrea Bonarini,et al.  Modeling enjoyment preference from physiological responses in a car racing game , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[43]  Yann LeCun,et al.  Comparing SVM and convolutional networks for epileptic seizure prediction from intracranial EEG , 2008, 2008 IEEE Workshop on Machine Learning for Signal Processing.

[44]  D. Tzovaras,et al.  Using Activity-Related Behavioural Features towards More Effective Automatic Stress Detection , 2012, PloS one.

[45]  K. Kallinen,et al.  Phasic Emotional Reactions to Video Game Events: A Psychophysiological Investigation , 2006 .

[46]  S M Pincus,et al.  Approximate entropy as a measure of system complexity. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[47]  Andrea Bonarini,et al.  Enjoyment recognition from physiological data in a car racing game , 2010, AFFINE '10.

[48]  Georgios N. Yannakakis,et al.  Generic Physiological Features as Predictors of Player Experience , 2011, ACII.

[49]  Georgios N. Yannakakis,et al.  Ranking vs. Preference: A Comparative Study of Self-reporting , 2011, ACII.

[50]  Georgios N. Yannakakis,et al.  Extending neuro-evolutionary preference learning through player modeling , 2010, Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games.

[51]  Yves Chauvin,et al.  Backpropagation: theory, architectures, and applications , 1995 .

[52]  Georgios N. Yannakakis Preference learning for affective modeling , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[53]  K. H. Kim,et al.  Emotion recognition system using short-term monitoring of physiological signals , 2004, Medical and Biological Engineering and Computing.

[54]  Jürgen Schmidhuber,et al.  Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction , 2011, ICANN.

[55]  Georgios N. Yannakakis,et al.  Entertainment capture through heart rate activity in physical interactive playgrounds , 2008, User Modeling and User-Adapted Interaction.

[56]  Rama Chellappa,et al.  Discriminant analysis of principal components for face recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[57]  Georgios N. Yannakakis,et al.  Genetic search feature selection for affective modeling: a case study on reported preferences , 2010, AFFINE '10.

[58]  Yoshua Bengio,et al.  On the Expressive Power of Deep Architectures , 2011, ALT.

[59]  Marc'Aurelio Ranzato,et al.  Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Michele A. Parker,et al.  Relationship of Heart Rate Variability to Parasympathetic Effect , 2001, Circulation.

[61]  Yanjun Qi,et al.  Learning to rank with (a lot of) word features , 2010, Information Retrieval.

[62]  Douglas Eck,et al.  Temporal Pooling and Multiscale Learning for Automatic Annotation and Ranking of Music Audio , 2011, ISMIR.

[63]  Georgios N. Yannakakis,et al.  Mining multimodal sequential patterns: a case study on affect detection , 2011, ICMI '11.

[64]  Andrea Kleinsmith,et al.  Affective Body Expression Perception and Recognition: A Survey , 2013, IEEE Transactions on Affective Computing.

[65]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[66]  Masakazu Matsugu,et al.  Subject independent facial expression recognition with robust face detection using a convolutional neural network , 2003, Neural Networks.

[67]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[68]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[69]  Georgios N. Yannakakis,et al.  Entertainment modeling through physiology in physical play , 2008, Int. J. Hum. Comput. Stud..