Multimodal-multisensor affect detection

Imagine you are interested in analyzing the emotional responses of a person in some interaction context (i.e., with computer software, a robot, in a classroom, on the subway). You could simply ask the person to self-report his or her felt emotion using a questionnaire, a valence-arousal grid [Russell et al. 1989], a self-assessment manikin [Bradley and Lang 1994], or some such measurement instrument. Or you could ask trained humans to observe the person and provide emotion judgments [Ocumpaugh et al. 2015]. You could also record audio/video of the interaction and have trained coders annotate the videos for visible emotion at some later time. You can even use computer vision techniques to obtain automatic estimates of facial expressions in the videos [Girard et al. 2015]. Or you may be interested in the person’s physiological responses and can use a variety of sensors to collect these data.

[1]  Fabien Ringeval,et al.  AVEC'19: Audio/Visual Emotion Challenge and Workshop , 2019, ACM Multimedia.

[2]  Alessandro Vinciarelli,et al.  Multimodal analysis of social signals , 2018, The Handbook of Multimodal-Multisensor Interfaces, Volume 2.

[3]  Mehdi Ammi,et al.  How do users perceive multimodal expressions of affects? , 2018, The Handbook of Multimodal-Multisensor Interfaces, Volume 2.

[4]  Rafael A. Calvo,et al.  Automated Detection of Engagement Using Video-Based Estimation of Facial Expressions and Heart Rate , 2017, IEEE Transactions on Affective Computing.

[5]  Ryan Shaun Joazeiro de Baker,et al.  Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms , 2016, TIIS.

[6]  Ajay B. Satpute,et al.  The Brain Basis of Positive and Negative Affect: Evidence from a Meta-Analysis of the Human Neuroimaging Literature. , 2016, Cerebral cortex.

[7]  Sidney K. D'Mello,et al.  On the Influence of an Iterative Affect Annotation Approach on Inter-Observer and Self-Observer Reliability , 2016, IEEE Transactions on Affective Computing.

[8]  Rosalind W. Picard,et al.  Multiple Arousal Theory and Daily-Life Electrodermal Activity Asymmetry , 2016 .

[9]  Stefan Wermter,et al.  Multimodal emotional state recognition using sequence-dependent deep hierarchical features , 2015, Neural Networks.

[10]  Laszlo A. Jeni,et al.  Spontaneous facial expression in unscripted social interactions can be measured automatically , 2015, Behavior research methods.

[11]  Jean-Philippe Thiran,et al.  Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data , 2015, Pattern Recognit. Lett..

[12]  Stefan Winkler,et al.  Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning , 2015, ICMI.

[13]  Ryan Shaun Joazeiro de Baker,et al.  Accuracy vs. Availability Heuristic in Multimodal Affect Detection in the Wild , 2015, ICMI.

[14]  Qin Jin,et al.  Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks , 2015, AVEC@ACM Multimedia.

[15]  Dongmei Jiang,et al.  Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks , 2015, AVEC@ACM Multimedia.

[16]  Ting Dang,et al.  An Investigation of Annotation Delay Compensation and Output-Associative Fusion for Multimodal Continuous Emotion Prediction , 2015, AVEC@ACM Multimedia.

[17]  Alessandro Rozza,et al.  Multimodal Affective Analysis combining Regularized Linear Regression and Boosted Regression Trees , 2015, AVEC@ACM Multimedia.

[18]  Patrick Cardinal,et al.  ETS System for AV+EC 2015 Challenge , 2015, AVEC@ACM Multimedia.

[19]  Patrick Thiam,et al.  Ensemble Methods for Continuous Affect Recognition: Multi-modality, Temporality, and Challenges , 2015, AVEC@ACM Multimedia.

[20]  Ya Li,et al.  Long Short Term Memory Recurrent Neural Network based Multimodal Dimensional Emotion Recognition , 2015, AVEC@ACM Multimedia.

[21]  Fabien Ringeval,et al.  AV+EC 2015: The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data , 2015, AVEC@ACM Multimedia.

[22]  Jacqueline Kory Westlund,et al.  Motion Tracker: Camera-Based Monitoring of Bodily Movements Using Motion Silhouettes , 2015, PloS one.

[23]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[24]  Ryan Shaun Joazeiro de Baker,et al.  Automatic Detection of Learning-Centered Affective States in the Wild , 2015, IUI.

[25]  Sidney K. D'Mello,et al.  A Review and Meta-Analysis of Multimodal Affect Detection Systems , 2015, ACM Comput. Surv..

[26]  Jiebo Luo,et al.  Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing , 2015, AAAI.

[27]  Sidney K. D'Mello,et al.  Affect Elicitation for Affective Computing , 2015 .

[28]  J. Gratch,et al.  The Oxford Handbook of Affective Computing , 2014 .

[29]  Shiguang Shan,et al.  Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild , 2014, ICMI.

[30]  Thomas F. Quatieri,et al.  Vocal and Facial Biomarkers of Depression based on Motor Incoordination and Timing , 2014, AVEC '14.

[31]  B. Mesquita,et al.  Emotions in Context: A Sociodynamic Model of Emotions , 2014 .

[32]  L. F. Barrett.  The Conceptual Act Theory: A Précis , 2014 .

[33]  Jessica L. Tracy,et al.  An Evolutionary Approach to Understanding Distinct Emotions , 2014 .

[34]  Mohammad Soleymani,et al.  Continuous emotion detection using EEG signals and facial expressions , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[35]  Kristy Elizabeth Boyer,et al.  Predicting Learning and Affect from Multimodal Data Streams in Task-Oriented Tutorial Dialogue , 2014, EDM.

[36]  Qiang Ji,et al.  Hybrid video emotional tagging using users’ EEG and video content , 2014, Multimedia Tools and Applications.

[37]  Razvan Pascanu,et al.  Combining modality specific deep neural networks for emotion recognition in video , 2013, ICMI '13.

[38]  Wijnand A. IJsselsteijn,et al.  Machines Outperform Laypersons in Recognizing Emotions Elicited by Autobiographical Recollection , 2013, Hum. Comput. Interact..

[39]  Arthur C. Graesser,et al.  Unimodal and Multimodal Human Perceptionof Naturalistic Non-Basic Affective Statesduring Human-Computer Interactions , 2013, IEEE Transactions on Affective Computing.

[40]  Valerie J. Shute,et al.  Assessment and Learning of Qualitative Physics in Newton's Playground , 2013, AIED.

[41]  Friedhelm Schwenker,et al.  Kalman Filter Based Classifier Fusion for Affective State Recognition , 2013, MCS.

[42]  Björn W. Schuller,et al.  YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual Context , 2013, IEEE Intelligent Systems.

[43]  Verónica Pérez-Rosas,et al.  Multimodal Sentiment Analysis of Spanish Online Videos , 2013, IEEE Intelligent Systems.

[44]  Rafael A. Calvo,et al.  Beyond the basic emotions: what should affective computing compute? , 2013, CHI Extended Abstracts.

[45]  Fabien Ringeval,et al.  Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[46]  Peter Robinson,et al.  Dimensional affect recognition using Continuous Conditional Random Fields , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[47]  Björn W. Schuller,et al.  LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework , 2013, Image Vis. Comput..

[48]  V. Štruc,et al.  Towards Efficient Multi-Modal Emotion Recognition , 2013 .

[49]  Uwe Handmann,et al.  Attention and Emotion Based Adaption of Dialog Systems , 2013 .

[50]  N. Srinivasan,et al.  Role of affect in decision making. , 2013, Progress in brain research.

[51]  Kristen A. Lindquist,et al.  The hundred-year emotion war: are emotions natural kinds or psychological constructions? Comment on Lench, Flores, and Bench (2011). , 2013, Psychological bulletin.

[52]  Heather C. Lench,et al.  Searching for evidence, not a war: reply to Lindquist, Siegel, Quigley, and Barrett (2013). , 2013, Psychological bulletin.

[53]  Rafael A. Calvo,et al.  Classification of affects using head movement, skin color features and physiological signals , 2012, 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[54]  Rafael A. Calvo,et al.  Combining Classifiers in Multimodal Affect Detection , 2012, AusDM.

[55]  Rohit Kumar,et al.  Ensemble of SVM trees for multimodal emotion recognition , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[56]  Friedhelm Schwenker,et al.  Fusion of Fragmentary Classifier Decisions for Affective State Recognition , 2012, MPRSS.

[57]  Yunde Jia,et al.  Audio-visual emotion recognition with boosted coupled HMM , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[58]  Arman Savran,et al.  Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering , 2012, ICMI '12.

[59]  Björn W. Schuller,et al.  AVEC 2012: the continuous audio/visual emotion challenge , 2012, ICMI '12.

[60]  Kristen A. Lindquist,et al.  The brain basis of emotion: A meta-analytic review , 2012, Behavioral and Brain Sciences.

[61]  Björn W. Schuller,et al.  Context-Sensitive Learning for Enhanced Audiovisual Emotion Classification , 2012, IEEE Transactions on Affective Computing.

[62]  Thierry Pun,et al.  Multimodal Emotion Recognition in Response to Videos , 2012, IEEE Transactions on Affective Computing.

[63]  Klaus R. Scherer,et al.  A psycho-ethological approach to social signal processing , 2012, Cognitive Processing.

[64]  Chung-Hsien Wu,et al.  Error Weighted Semi-Coupled Hidden Markov Model for Audio-Visual Emotion Recognition , 2012, IEEE Transactions on Multimedia.

[65]  Jeong-Sik Park,et al.  Music-aided affective interaction between human and service robot , 2012, EURASIP Journal on Audio, Speech, and Music Processing.

[66]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[67]  Roddy Cowie,et al.  Tracing Emotion: An Overview , 2012, Int. J. Synth. Emot..

[68]  Maja Pantic,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING , 2022 .

[69]  Chung-Hsien Wu,et al.  Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels , 2015, IEEE Transactions on Affective Computing.

[70]  Johannes Wagner,et al.  A systematic discussion of fusion techniques for multi-modal affect recognition tasks , 2011, ICMI '11.

[71]  Guillaume Chanel,et al.  Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[72]  Sidney D'Mello,et al.  The half-life of cognitive-affective states during complex learning , 2011, Cognition & emotion.

[73]  Dongmei Jiang,et al.  Audio Visual Emotion Recognition Based on Triple-Stream Dynamic Bayesian Network Models , 2011, ACII.

[74]  Björn W. Schuller,et al.  AVEC 2011-The First International Audio/Visual Emotion Challenge , 2011, ACII.

[75]  Björn W. Schuller,et al.  Recognizing Affect from Linguistic Information in 3D Continuous Space , 2011, IEEE Transactions on Affective Computing.

[76]  Johannes Wagner,et al.  Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data , 2011, IEEE Transactions on Affective Computing.

[77]  Ira J. Roseman.  Emotional Behaviors, Emotivational Goals, Emotion Strategies: Multiple Levels of Organization Integrate Variable and Consistent Responses , 2011 .

[78]  Friedhelm Schwenker,et al.  Multimodal Emotion Classification in Naturalistic User Behavior , 2011, HCI.

[79]  Hatice Gunes,et al.  Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space , 2011, IEEE Transactions on Affective Computing.

[80]  Kaoru Hirota,et al.  Emotion recognition based on human gesture and speech information using RT middleware , 2011, 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011).

[81]  Léon J. M. Rothkrantz,et al.  Emotion recognition using bimodal data fusion , 2011, CompSysTech '11.

[82]  James J. Gross,et al.  Emotion Generation and Emotion Regulation: One or Two Depends on Your Point of View , 2011, Emotion review : journal of the International Society for Research on Emotion.

[83]  C. Izard.  The Many Meanings/Aspects of Emotion: Definitions, Functions, Activation, and Regulation , 2010 .

[84]  J. Coan.  Emergent Ghosts of the Emotion Machine , 2010 .

[85]  L. Camras,et al.  Emotional Facial Expressions in Infancy , 2010 .

[86]  Rosalind W. Picard.  Affective Computing: From Laughter to IEEE , 2010 .

[87]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[88]  Björn W. Schuller,et al.  On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues , 2009, Journal on Multimodal User Interfaces.

[89]  K. Scherer.  The dynamic architecture of emotion: Evidence for the component process model , 2009 .

[90]  Ilse Ravyse,et al.  Audio-Visual Emotion Recognition Based on a DBN Model with Constrained Asynchrony , 2009, 2009 Fifth International Conference on Image and Graphics.

[91]  Cristina Conati,et al.  Empirically building and evaluating a probabilistic model of user affect , 2009, User Modeling and User-Adapted Interaction.

[92]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  B. Mesquita,et al.  The experience of emotion. , 2007, Annual review of psychology.

[94]  K. Scherer,et al.  The World of Emotions is not Two-Dimensional , 2007, Psychological science.

[95]  C. Izard.  Basic Emotions, Natural Kinds, Emotion Schemas, and a New Paradigm , 2007, Perspectives on psychological science : a journal of the Association for Psychological Science.

[96]  John J. B. Allen,et al.  The handbook of emotion elicitation and assessment , 2007 .

[97]  L. F. Barrett.  Are Emotions Natural Kinds? , 2006, Perspectives on psychological science : a journal of the Association for Psychological Science.

[98]  Marc D. Lewis.  Bridging emotion theory and neurobiology through dynamic systems modeling , 2005, Behavioral and Brain Sciences.

[99]  Qiang Ji,et al.  Active affective State detection and user assistance with dynamic bayesian networks , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[100]  Agneta H. Fischer,et al.  Emotion in Social Relations: Cultural, Group, and Interpersonal Processes , 2004 .

[101]  Anne Parkhill,et al.  Searching for the Evidence , 2004, Australasian chiropractic & osteopathy : journal of the Chiropractic & Osteopathic College of Australasia.

[102]  C. Don.  Looking for Spinoza: Joy, Sorrow, and the Feeling Brain , 2004 .

[103]  J. Russell,et al.  Facial and vocal expressions of emotion. , 2003, Annual review of psychology.

[104]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[105]  G. Loewenstein,et al.  The role of affect in decision making. , 2003 .

[106]  J. Russell.  Core affect and the psychological construction of emotion. , 2003, Psychological review.

[107]  D. R. Lehman,et al.  What's wrong with cross-cultural comparisons of subjective Likert scales?: The reference-group effect. , 2002, Journal of personality and social psychology.

[108]  Hillary Anger Elfenbein,et al.  Is there an in-group advantage in emotion recognition? , 2002, Psychological bulletin.

[109]  Hillary Anger Elfenbein,et al.  On the universality and cultural specificity of emotion recognition: a meta-analysis. , 2002, Psychological bulletin.

[110]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[111]  J. Lerner,et al.  Beyond valence: Toward a model of emotion-specific influences on judgement and choice , 2000 .

[112]  J. Krosnick,et al.  Survey research. , 1999, Annual review of psychology.

[113]  Nirbhay N. Singh,et al.  Facial Expressions of Emotion , 1998 .

[114]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[115]  W. Ruch.  Will the real relationship between facial expression and affective experience please stand up: The case of exhilaration , 1995 .

[116]  P. Ekman,et al.  Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. , 1994, Psychological bulletin.

[117]  C. Izard.  Innate and universal facial expressions: evidence from developmental and cross-cultural research. , 1994, Psychological bulletin.

[118]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[119]  D. Guttfreund,et al.  The effects of written autobiographical recollection induction procedures on mood. , 1993, Journal of clinical psychology.

[120]  P. Ekman.  An argument for basic emotions , 1992 .

[121]  G. A. Mendelsohn,et al.  Affect grid : A single-item scale of pleasure and arousal , 1989 .

[122]  Ralph L. Rosnow,et al.  Essentials of Behavioral Research: Methods and Data Analysis , 1984 .

[123]  P. Ekman,et al.  Facial Expressions of Emotion , 1979 .

[124]  W. James,et al.  What Is an Emotion , 1977 .

[125]  C. Darwin,et al.  The Expression of the Emotions in Man and Animals , 1872 .