A Multimodal Database for Affect Recognition and Implicit Tagging

MAHNOB-HCI is a multimodal database recorded in response to affective stimuli with the goal of emotion recognition and implicit tagging research. A multimodal setup was arranged for synchronized recording of face videos, audio signals, eye gaze data, and peripheral/central nervous system physiological signals. Twenty-seven participants from both genders and different cultural backgrounds participated in two experiments. In the first experiment, they watched 20 emotional videos and self-reported their felt emotions using arousal, valence, dominance, and predictability as well as emotional keywords. In the second experiment, short videos and images were shown once without any tag and then with correct or incorrect tags. Agreement or disagreement with the displayed tags was assessed by the participants. The recorded videos and bodily responses were segmented and stored in a database. The database is made available to the academic community via a web-based system. The collected data were analyzed and single modality and modality fusion results for both emotion recognition and implicit tagging experiments are reported. These results show the potential uses of the recorded modalities and the significance of the emotion elicitation protocol.

[1]  Maja Pantic,et al.  Particle filtering with factorized likelihoods for tracking facial features , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[2]  Hatice Gunes,et al.  Static vs. dynamic modeling of human nonverbal behavior from multiple cues and modalities , 2009, ICMI-MLMI '09.

[3]  J. Russell.  Culture and the categorization of emotions. , 1991, Psychological bulletin.

[4]  P. Ekman,et al.  Approach-withdrawal and cerebral asymmetry: emotional expression and brain physiology. I. , 1990, Journal of personality and social psychology.

[5]  Shrikanth S. Narayanan,et al.  The Vera am Mittag German audio-visual emotional speech database , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[6]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[7]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[8]  Manuel Menezes de Oliveira Neto,et al.  Photorealistic models for pupil light reflex and iridal pattern deformation , 2009, TOGS.

[9]  R. Davidson.  Affective neuroscience and psychophysiology: toward a synthesis. , 2003, Psychophysiology.

[10]  M. Bradley,et al.  The pupil as a measure of emotional arousal and autonomic activation. , 2008, Psychophysiology.

[11]  David Sander,et al.  A systems approach to appraisal mechanisms in emotion , 2005, Neural Networks.

[12]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[13]  Mohammad Soleymani,et al.  A collaborative personalized affective video retrieval system , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[14]  R. Adolphs,et al.  Dissociable neural systems for recognizing emotions , 2003, Brain and Cognition.

[15]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  H. Bouma,et al.  Hippus of the pupil: periods of slow oscillations of unknown origin. , 1971, Vision research.

[17]  R. Cowie,et al.  A new emotion database: considerations, sources and scope , 2000 .

[18]  Kostas Karpouzis,et al.  The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data , 2007, ACII.

[19]  R. Davidson,et al.  Prefrontal Brain Asymmetry: A Biological Substrate of the Behavioral Approach and Inhibition Systems , 1997 .

[20]  A. Damasio,et al.  Basic emotions are associated with distinct patterns of cardiorespiratory activity. , 2006, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[21]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[22]  M. Pantic,et al.  Induced Disgust , Happiness and Surprise : an Addition to the MMI Facial Expression Database , 2010 .

[23]  David A. Forsyth,et al.  Generalizing motion edits with Gaussian processes , 2009, ACM Trans. Graph..

[24]  A. Freitas-Magalhães.  Facial Expression of Emotion , 2012 .

[25]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[26]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  A. Schaefer,et al.  Please Scroll down for Article Cognition & Emotion Assessing the Effectiveness of a Large Database of Emotion-eliciting Films: a New Tool for Emotion Researchers , 2022 .

[28]  M. Bradley,et al.  Looking at pictures: affective, facial, visceral, and behavioral reactions. , 1993, Psychophysiology.

[29]  Jennifer Healey,et al.  Detecting stress during real-world driving tasks using physiological sensors , 2005, IEEE Transactions on Intelligent Transportation Systems.

[30]  Jon D. Morris.  Observations: SAM: The Self-Assessment Manikin An Efficient Cross-Cultural Measurement Of Emotional Response 1 , 1995 .

[31]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Maja Pantic,et al.  Implicit human-centered tagging [Social Sciences] , 2009, IEEE Signal Process. Mag..

[33]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[34]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[35]  Mohammad Soleymani,et al.  Short-term emotion assessment in a recall paradigm , 2009, Int. J. Hum. Comput. Stud..

[36]  Maja Pantic,et al.  Is this joke really funny? judging the mirth by audiovisual laughter analysis , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[37]  R. A. Mcfarland.  Relationship of skin temperature changes to the emotions accompanying music , 1985, Biofeedback and self-regulation.

[38]  K. Scherer,et al.  The World of Emotions is not Two-Dimensional , 2007, Psychological science.

[39]  Veikko Surakka,et al.  Pupil size variation as an indication of affective processing , 2003, Int. J. Hum. Comput. Stud..

[40]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[41]  Jun Jiao,et al.  Implicit image tagging via facial information , 2010, SSPW '10.

[42]  Maja Pantic,et al.  The SEMAINE corpus of emotionally coloured character interactions , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[43]  Maja Pantic,et al.  Cost-effective solution to synchronised audio-visual data capture using multiple sensors , 2011, Image Vis. Comput..

[44]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[45]  R. Mccraty,et al.  The effects of emotions on short-term power spectrum analysis of heart rate variability . , 1995, The American journal of cardiology.