ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors

We present ASCERTAIN—a multimodal databa<bold>AS</bold>e for impli<bold>C</bold>it p<bold>ER</bold>sonali<bold> T</bold>y and <bold>A</bold>ffect recognit<bold>I</bold>o<bold>N</bold> using commercial physiological sensors. To our knowledge, ASCERTAIN is the first database to connect <italic>personality traits</italic> and <italic>emotional states </italic> via <italic>physiological responses</italic>. ASCERTAIN contains big-five personality scales and emotional self-ratings of 58 users along with their Electroencephalogram (EEG), Electrocardiogram (ECG), Galvanic Skin Response (GSR) and facial activity data, recorded using off-the-shelf sensors while viewing affective movie clips. We first examine relationships between users’ affective ratings and personality scales in the context of prior observations, and then study linear and non-linear physiological correlates of emotion and personality. Our analysis suggests that the emotion-personality relationship is better captured by non-linear rather than linear statistics. We finally attempt binary emotion and personality trait recognition using physiological features. Experimental results cumulatively confirm that personality differences are better revealed while comparing user responses to emotionally homogeneous videos, and above-chance recognition is achieved for both affective and personality dimensions.

[1]  N. Sebe,et al.  Emotion modulates eye movement patterns and subsequent memory for the gist and details of movie scenes. , 2014, Journal of vision.

[2]  Subramanian Ramanathan,et al.  On the relationship between head pose, social attention and personality prediction for unstructured and dynamic group interactions , 2013, ICMI '13.

[3]  G. Kraaykamp,et al.  Personality, media preferences, and cultural participation , 2005 .

[4]  Ioannis Patras,et al.  Fusion of facial expressions and EEG for implicit affective tagging , 2013, Image Vis. Comput..

[5]  Susan Sullivan,et al.  What’s good for the goose is not good for the gander: Age and gender differences in scanning emotion faces , 2017, The journals of gerontology. Series B, Psychological sciences and social sciences.

[6]  Maja Pantic,et al.  Web-based database for facial expression analysis , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[7]  Nicu Sebe,et al.  Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents , 2010, Multimedia Tools and Applications.

[8]  P. Costa,et al.  Influence of extraversion and neuroticism on subjective well-being: happy and unhappy people. , 1980, Journal of personality and social psychology.

[9]  R. Bornstein.  Exposure and affect: Overview and meta-analysis of research, 1968–1987. , 1989 .

[10]  Weiting Ng,et al.  Clarifying the relation between neuroticism and positive emotions , 2009 .

[11]  V. Pascalis,et al.  Personality, event-related potential (ERP) and heart rate (HR) in emotional word processing ☆ , 2004 .

[12]  Kostas Karpouzis,et al.  The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data , 2007, ACII.

[13]  Michael Eid,et al.  Why extraverts are happier than introverts: the role of mood regulation. , 2006, Journal of personality.

[14]  Shrikanth S. Narayanan,et al.  Toward detecting emotions in spoken dialogs , 2005, IEEE Transactions on Speech and Audio Processing.

[15]  Loïc Kessous,et al.  Modeling naturalistic affective states via facial and vocal expressions recognition , 2006, ICMI '06.

[16]  Alex Pentland,et al.  Sensible Organizations: Technology and Methodology for Automatically Measuring Organizational Behavior , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[18]  Christine L. Lisetti,et al.  Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals , 2004, EURASIP J. Adv. Signal Process..

[19]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Nicu Sebe,et al.  Emotion Recognition Based on Joint Visual and Audio Cues , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[21]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[22]  Jun Wang,et al.  A 3D facial expression database for facial behavior research , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[23]  Ravi Kothuri,et al.  Physiological Correlates of the Big 5: Autonomic Responses to Video Presentations , 2013, Applied psychophysiology and biofeedback.

[24]  Karla R. Hamlen,et al.  The Effects of Familiarity and Audiovisual Stimuli on Preference for Classical Music , 2006 .

[25]  Filip De Fruyt,et al.  Linking openness and intellect in childhood and adulthood , 1998 .

[26]  G. Stenberg,et al.  Personality and the EEG: Arousal and emotional arousability , 1992 .

[27]  Alessandro Vinciarelli,et al.  A Survey of Personality Computing , 2014, IEEE Transactions on Affective Computing.

[28]  Nicu Sebe,et al.  Friends don't lie: inferring personality traits from social network structure , 2012, UbiComp.

[29]  Oscar García,et al.  Relationships among extraversion, openness to experience, and sensation seeking , 2003 .

[30]  Janne Heikkilä,et al.  Predicting the Valence of a Scene from Observers’ Eye Movements , 2015, PloS one.

[31]  J. Gross,et al.  Emotion elicitation using films , 1995 .

[32]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[33]  M. Koyuncu,et al.  Evaluation of International Affective Picture System (IAPS) ratings in an athlete population and its relations to personality , 2010 .

[34]  Colin Studholme,et al.  An overlap invariant entropy measure of 3D medical image alignment , 1999, Pattern Recognit..

[35]  Elizabeth G. Kehoe,et al.  Personality modulates the effects of emotional arousal and valence on brain activation. , 2012, Social cognitive and affective neuroscience.

[36]  Elmar Nöth,et al.  How to find trouble in communication , 2003, Speech Commun..

[37]  Elisa Ricci,et al.  Space speaks: towards socially and personality aware visual surveillance , 2010, MPVA '10.

[38]  Alexander Toet,et al.  Neuroticism, Extraversion, Conscientiousness and Stress: Physiological Correlates , 2015, IEEE Transactions on Affective Computing.

[39]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[40]  Bryant Paul,et al.  Effects of Personality Types on the Use of Television Genre , 2007 .

[41]  Elmar Nöth,et al.  “You Stupid Tin Box” - Children Interacting with the AIBO Robot: A Cross-linguistic Emotional Speech Corpus , 2004, LREC.

[42]  Shuicheng Yan,et al.  Don't ask me what i'm like, just watch and listen , 2012, ACM Multimedia.

[43]  Qiang Ji,et al.  Video Affective Content Analysis: A Survey of State-of-the-Art Methods , 2015, IEEE Transactions on Affective Computing.

[44]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[45]  Rong Hu,et al.  A comparative user study on rating vs. personality quiz based preference elicitation methods , 2009, IUI.

[46]  Nicu Sebe,et al.  Inference of personality traits and affect schedule by analysis of spontaneous reactions to affective videos , 2015, 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[47]  Subramanian Ramanathan,et al.  DECAF: MEG-Based Multimodal Database for Decoding Affective Physiological Responses , 2015, IEEE Transactions on Affective Computing.

[48]  Subramanian Ramanathan,et al.  Connecting Meeting Behavior with Extraversion—A Systematic Study , 2012, IEEE Transactions on Affective Computing.

[49]  Marilyn A. Walker,et al.  Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text , 2007, J. Artif. Intell. Res..

[50]  C. Witvliet,et al.  Play it again Sam: Repeated exposure to emotionally evocative music polarises liking and smiling responses, and influences other affective reports, facial EMG, and heart rate , 2007 .

[51]  Subramanian Ramanathan,et al.  SALSA: A Novel Dataset for Multimodal Group Behavior Analysis , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  N. Kuiper,et al.  Individual differences in the experience of emotions. , 1997, Clinical psychology review.